Jump to content

User talk:DPLA bot

Add topic
From Wikimedia Commons, the free media repository
Latest comment: 1 hour ago by Dominic in topic A glitch

strange dupe-tagging

[edit]
Latest comment: 8 days ago54 comments9 people in discussion

Hi DPLA_bot, today you (resp. your bot) has speedy-tagged a number of files as duplicates, which - wrt their content - are not duplicates at all.

and many more. Is this really intentional? --Túrelio (talk) 07:59, 20 May 2026 (UTC)Reply

@Túrelio: That bot only do shit at the moment, moving files like File:Ronald Reagan and Douglas Ginsburg.jpg, uploaded by User, Change the file description and deleted all categories. Adding or updating the file description might be acceptable, but renaming the file is rather questionable, as is deleting categories – totally unnecessary, yes, even vandalism. זיו「Ziv」 • For love letters and other notes 17:11, 20 May 2026 (UTC)Reply
Addendum: The license is now also incorrect. זיו「Ziv」 • For love letters and other notes 17:43, 20 May 2026 (UTC)Reply
Yes, I’ve been working on cleaning this up as soon as I realized. I appreciate your patience in allowing me resolve it. I undid all the duplicate tags that shouldn’t have gone out, and will sort out the rest, too. I understand this is frustrating, but it’s certainly not ill-intentioned. The bot runs at high volume. Dominic (talk) 17:49, 20 May 2026 (UTC)Reply
Thank you very much for your answer @Dominic. I don't know if there are any other files like the one I mentioned above; I just noticed just this one. If so, please correct them, thank you. Best regards, זיו「Ziv」 • For love letters and other notes 18:04, 20 May 2026 (UTC)Reply
Also, I didn't mean to be short above, just trying to work quickly. I want to give a fuller explanation later, but I can tell already that what happened was I had some test code I was working on that then ran overnight before I realized it was not the regular code. My first priority is just rolling back any unintended edits. Dominic (talk) 18:09, 20 May 2026 (UTC)Reply
same thing happened to File:Nancy Reagan waves while visiting the Statue of Liberty in New York City.jpg Surajr7 (talk) 18:21, 20 May 2026 (UTC)Reply
I agree about the renaming, why does the file name need to have the DPLA id in it? It just clutters the file name and makes it extra lengthy for no benefit. Traumnovelle (talk) 01:15, 27 May 2026 (UTC)Reply
@Traumnovelle: Thanks for asking! There are a few different benefits. One is that the institutions who provide these files are maintaining their metadata over time, and some times corrections or changes are made. The goal of our project is synchonize that dataset with the uploads on Commons so that they are also maintained here. Standardizing the image titles and metadata allows them to be maintained by bot. The only images the bot will touch are ones that are exact hash matches for the file from the institution's own catalog, meaning someone uploaded the file to Commons from the institution exactly as is. Sometimes they are uploaded here either with a title copied from Flickr, which wasn't really selected by the user anyway, and comes from the institution ultimately, or it might be a user-generated title which is sometimes not descriptive, and not trackable. I know what you mean about adding length to some file names, but it's not useless or for vanity, it's for a real purpose! Dominic (talk) 04:46, 27 May 2026 (UTC)Reply
@Traumnovelle: FWIW, working in categories with a lot of archival materials, I find it useful to know at a glance which ones came into Commons via DPLA bot. Sets my expectations very clearly for what will be the strengths and weaknesses of the metadata (in the broad sense of the latter). - Jmabel ! talk 14:31, 27 May 2026 (UTC)Reply
File names such as 'Woldruck - DPLA' would convey the same information in that regard. Traumnovelle (talk) 20:14, 27 May 2026 (UTC)Reply
There is more to it than that, though. There are over 10 million files from DPLA, so unique names are essential, and also the IDs are the only way would could associate them with the source metadata and maintain them. Dominic (talk) 03:53, 28 May 2026 (UTC)Reply
@Dominic i think you should make use of com:sdc instead of using the filename which is pretty damn useless and unimportant.
if necessary, make a new property like d:Wikidata:Property proposal/Flickr Photo ID did.
take a look at how flickr backfilled their metadata User:FlickypediaBackfillrBot.
https://commons.wikimedia.org/w/index.php?diff=1222566342 you removed some commons users' work, which is not from the institutions you are working for. that should be avoided. RoyZuo (talk) 09:08, 30 May 2026 (UTC)Reply
there's actually already DPLA ID (P760). you should write that to sdc instead of doing anything with filename which is a maintenance burden and not stable. RoyZuo (talk) 09:13, 30 May 2026 (UTC)Reply
@RoyZuo: Please add back any descriptions that the bot mistakenly deleted, like this was once the media of the day etc.. זיו「Ziv」 • For love letters and other notes 06:41, 7 June 2026 (UTC)Reply
Hey @RoyZuo and @Ziv, thanks for flagging this and for the fix on that one. You're right, it was a real regression: our new title-drift metadata-rescue code is preserving other user-added content like licenses, "image extracted" templates, categories, etc., but assessment-class templates like MOTD weren't accounted for and got dropped. I scanned all 11,588 of the bot's rescue edits since that code shipped on May 18, and this Chiang file was the only one that lost such a template — and Ziv had already restored it manually before I got there. The fix is now live (dpla/ingest-wikimedia#277) and covers Media of the day, Picture of the day, Featured/Quality/Valued picture, and the {{Assessments}} bundle going forward. Dominic (talk) 21:29, 7 June 2026 (UTC)Reply
a user contributed description in ukrainian. RoyZuo (talk) 22:17, 7 June 2026 (UTC)Reply
This is more complicated to parse out, but I am looking into it. Dominic (talk) 22:52, 7 June 2026 (UTC)Reply
@Ziv and RoyZuo: , just to follow up, I did a full sweep and found about 30 files that were affected. I have added them back in, with a series of edits like this one. Let me know if you see any other issues. Dominic (talk) 05:02, 12 June 2026 (UTC)Reply
@Dominic, looks good for me. Thank you. זיו「Ziv」 • For love letters and other notes 06:05, 12 June 2026 (UTC)Reply
@Dominic thx a lot for the recent update using sdc. it looks very neat.
one potential problem from your file moves. if a file is audio or video, it may have com:timedtext, which i think are not automatically moved. i'm not sure if your bot has accounted for this situation. if not, you might wanna check these files and move the timedtext accordingly. RoyZuo (talk) 16:57, 14 June 2026 (UTC)Reply

There are a couple of things going on here to explain:

  1. The {{Duplicate}} tagging wasn't working as intended, and is, of course, only intended for non-controversial exact duplicate cleanup. I reverted them all, but sometimes images do get uploaded twice when a title or ID changes, so I will go back and make sure no actual duplicates are left in place.
  2. Sometimes we try to upload from a partner and find out that a Wikimedian already uploaded it. Or that we uploaded it before under a different name. In these cases, we want to maintain the metadata and title, so there is proper information and attribution, It might have been uploaded 10 years ago with minimal description. What I was testing was how we could do that cleanly. Overwriting valuable info like categories and copyright templates was not intentional, and is a new insight when we do implement anything like this.
  3. This was all test code not intended to be run without monitoring. Some of it was even unfinished. I wouldn't normally test like that, and never have. Please let me know if you catch anything I didn't mention above. Dominic (talk) 18:31, 20 May 2026 (UTC)Reply
Sorry, but this is still inaceptable. Like File:David Addington, Lucy Tutwiler, and Katie Wilson Wearing Body Armor at Hakim Compound in Red Zone, Baghdad - DPLA - e259228a1ef65e5fe7223a6f841c2e39.jpg. Old file name was good enough. The DPLA number can be included in the file description, but it doesn't have to be in the filename. Please stop moving correctly named files. זיו「Ziv」 • For love letters and other notes 20:21, 22 May 2026 (UTC)Reply
Addendum: Overwriting valid licenses is also a problem here. Originally, this was a Flickr upload with the license {{Flickr-no known copyright restrictions}}, now we have {{PD-US}}, which states that the image is not in the public domain in some countries, which was previously the case. The correct license should be {{PD-USGov}}. The bot should check which files it has inserted an incorrect license into and adjust them accordingly. זיו「Ziv」 • For love letters and other notes 21:54, 22 May 2026 (UTC)Reply
I hear why you’re frustrated, so I just want to clarify a few things. The bot is operating under the direction of the National Archives itself. Whatever they previously uploaded to Flickr is not as authoritative as the current metadata and identifier. We are only touching files that are exact hash matches for the file in the current NARA catalog. When one is detected, I am renaming it to provide the accurate current title, metadata, and identifier, and link to the new catalog. Most of these are many years old. I am doing our best to do it the right way, such as leaving redirects in place and using the User:CommonsDelinker process so no links are broken. If there is a PD-USGov template, it is retained. The Flickr tag is not necessary or helpful for a file which is directly copied from the official catalog, and the Flickr-no known copyright restrictions doesn’t really add anything, certainly not somehow worth retaining over the exact copyright statement from the institution does. Dominic (talk) 22:22, 22 May 2026 (UTC)Reply
Okay thank you. זיו「Ziv」 • For love letters and other notes 05:39, 23 May 2026 (UTC)Reply
Hello @Dominic:
For your information: I added the Category:Files exempt from duplicate tagging to both files File:(Sherman case docket) - DPLA - 2e9d3ca5d4c772d9e3da903fe4812265 (page 1).jpg and File:(Sherman case docket) - DPLA - 2e9d3ca5d4c772d9e3da903fe4812265 (page 12).jpg. Otherwise, OptimusPrimeBot would recognize them as duplicates again and tag them accordingly. This prevents them from being deleted again. Best regards, זיו「Ziv」 • For love letters and other notes 16:04, 24 May 2026 (UTC)Reply
Same also for File:(Sherman case docket) - DPLA - 2e9d3ca5d4c772d9e3da903fe4812265 (page 2).jpg & File:(Sherman case docket) - DPLA - 2e9d3ca5d4c772d9e3da903fe4812265 (page 13).jpg זיו「Ziv」 • For love letters and other notes 16:22, 24 May 2026 (UTC)Reply
Unfortunately, it's happening again that the bot is now marking older user uploads as duplicates. Example: File:Advertisment for the St. Louis and St. Clair Ferry, July 4, 1842.jpg. That shouldn't happen. זיו「Ziv」 • For love letters and other notes 17:24, 21 June 2026 (UTC)Reply
Plus also a batch that tagged duplicates like this File:Distribution Department, Low Service Spot Pond Reservoir, verso, "height of camera objective - 6.5 feet; note that a rise of water level will expose a new but excellent front", Ston - DPLA - 01e81012a2a12fa66704075773b08b0c.jpg. זיו「Ziv」 • For love letters and other notes 17:41, 21 June 2026 (UTC)Reply
@Ziv: There's a lot of edge cases that keep cropping up because I am trying to manage a backlog of maintenance on files, some of which are many years old. Thanks for bringing File:Advertisment for the St. Louis and St. Clair Ferry, July 4, 1842.jpg to my attention. Here's the complicated path: (1) In 2017 Fae uploads this file, (2) Back in 2024, DPLA_bot uploads this exact same item, but for whatever reason (they are visually identical, so it wasn't a bug per se), the MHS image server at the time was serving the same image at a different resolution, so there was no duplicate collision, (3) today, the bot is re-running this collection and MHS's image server is back to giving the file it gave Fae in 2017, (4) since we can't move the old file to the new correct name, where a file already exists, the bot overwrites the version at the desired title and tags the old one as duplicate. The part of this that went wrong is there is supposed to be a path that rescues categories and other user contributions, but it didn't trigger here. This already works in other situations, such as when a hash duplicate already exists on Commons without the presence of a file at the desired title (like this). What I am going to do is kill the ongoing process, fix the bug to ensure the categories and templates are copied over before duplicate tagging in this type of situation too, and then apply that to the ones that were already tagged, so they can still be safely deleted. Dominic (talk) 19:10, 21 June 2026 (UTC)Reply
Also, the second issue with the template breaking is a situation where the file name has the = character, breaking template syntax. A simple escaping bug that can be fixed like this. I will apply the fix in the code as well. Dominic (talk) 19:24, 21 June 2026 (UTC)Reply
All of the bad = templates were fixed, like so. Dominic (talk) 20:37, 21 June 2026 (UTC)Reply
@Ziv: There were about 40 files, I think all from Fae, where the older contributions, such as categories, have been rescued. But ultimately, in these cases, tagging the older file was correct, since the corrected title with the DPLA ID already has a file at it; where two files exist, meaning the situation can't be resolved with a file move, one needs to be deleted be and should be the one at the wrong name (as long as user contributions are being preserved, as it all has been now). Dominic (talk) 20:37, 21 June 2026 (UTC)Reply
@Dominic: Thank you very much for the quick solution. However, I am not entirely satisfied for files uploaded by Fae or other users years before the DPLA bot uploaded a identical file. I liked the way you handled, for example with File:Nancy Reagan Waves While Visiting The Statue of Liberty in New York City - DPLA - 27f33ae983dc977c6096f75934bd706d.jpg. Renaming the files may be controversial, but user uploads should definitely be retained. Best regards, זיו「Ziv」 • For love letters and other notes 08:40, 22 June 2026 (UTC)Reply
@Ziv: Right, that would be my goal as well. The issue here is that there is both a file previously uploaded at another name and a file already at the new file name. Let me know if you have a better idea I can implement, because I am happy to work on it. The most straightforward way I think we could handle this that deletes the newer file over the older file but still achieves the desired file rename would be to upload the duplicate file to the desired file name for the purpose of then immediately tagging it as as a duplicate—which I think would cause some confusion as well. And then I'd still have to come back later and re-run them all to actually do the renames once the existing file was deleted. Or, and this also seemed a little controversial to do on my own initiative, I could move the existing file to a temporary name, which would allow me to move the old file over the new redirect, and then tag the file at the temp name as a duplicate (so I don't have to do a second run). That would be the optimal, in a way, though it would look messy to anyone who didn't understand what we're doing. Dominic (talk) 14:17, 22 June 2026 (UTC)Reply
Lemme show you something on File:Advertisment for the St. Louis and St. Clair Ferry, July 4, 1842.jpg. The bot now only needs to rename the file. זיו「Ziv」 • For love letters and other notes 20:20, 22 June 2026 (UTC)Reply
If that's a good idea, then I can handle Fae's other uploads the same way. זיו「Ziv」 • For love letters and other notes 20:22, 22 June 2026 (UTC)Reply
Or i can move the files to the DPLA name myself. Then you have absolutely nothing left to do. זיו「Ziv」 • For love letters and other notes 20:26, 22 June 2026 (UTC)Reply
Dang! Sorry for the copy paste error... But you see what I'm getting at. Because there aren't that many. זיו「Ziv」 • For love letters and other notes 20:31, 22 June 2026 (UTC)Reply
Did now the same with File:A.F. Bullard, Captain, 38th Massachusetts Infantry (Union). - DPLA - 249b96450fce5f80ca03d997111fd6ea.jpg, without producing a double extension. זיו「Ziv」 • For love letters and other notes 20:48, 22 June 2026 (UTC)Reply
Hello @Dominic.
Even without waiting for your reply, I continued working on this today. I really don't see why user uploads and version histories are being destroyed here just because DPLA thinks "only we have the right file and the right filename." In that sense, I will continue to add the DPLA description to the uploads and adjust the filename accordingly, but that doesn't mean that just because there's an exact duplicate in your database, other databases are useless. Regards, זיו「Ziv」 • For love letters and other notes 23:24, 23 June 2026 (UTC)Reply
@Ziv: Sorry for stepping away for a few days. I am happy for you to resolve these as you see fit. It's designed to self-repair anyway, if it needs to be re-run on a category. The only problem it can't resolve is when there are two versions of a file, since the bot cannot delete. As long as something gets deleted, the next pass would then have a clear path to to rename and/or add SDC, as needed. I noticed today that a fast-running batch added several thousand files to Category:Duplicate. It looks like they are all legitimate duplicates of the scenario we were talking about before, but I killed that running batch and will try to engineer a guard in the code to prefent it from adding to the backlog when the category is already large. Let me know if you want me to undo all the current {{Duplicate}} tags and add them back more slowly. Dominic (talk) 19:04, 29 June 2026 (UTC)Reply
@Dominic the "case law" on this for my bot is to not add new duplicate files when the category exceeds 190 files, to avoid stressing administrators with a sudden large amount of files to review. Would it be possible for you to use the same limit? vip (talk) 23:18, 29 June 2026 (UTC)Reply
The maintenance category is actually very large at the moment, though it has been even bigger—with over 10,000 files from the DPLA bot. It simply takes time, but whenever I have some, I delete several hundred a day. זיו「Ziv」 • For love letters and other notes 02:50, 30 June 2026 (UTC)Reply
Yes, but right now, my bot completely stops its duplicate check activity until the number of files goes down to 190. It was an explicit request from amidnistrators, so all bots shall behave the same. vip (talk) 13:25, 30 June 2026 (UTC)Reply
@Dominic: I noticed that @Yann: has now deleted a shitload of your duplicates without leaving redirects. So, I’m asking you: what do you prefer; simply deleting it or creating the redirects? זיו「Ziv」 • For love letters and other notes 15:17, 30 June 2026 (UTC)Reply
Yes, I deleted over 8,000 duplicates. And this is not the first time. Please check before uploading. Thanks, Yann (talk) 15:25, 30 June 2026 (UTC)Reply
Thanks a lot, Yann. Good work. It would normally take hours, if not days, to create a redirect every time (which is what I usually do, too, unless the duplicate points to a file that is subject to a deletion request or a speedy deletion request). זיו「Ziv」 • For love letters and other notes 15:33, 30 June 2026 (UTC)Reply
@Yann: : Please allow me one criticism, however. The DPLA version of File:South Korean President Chun Doo Hwan meets U.S. President Ronald Reagan in the Oval Office (C439-13).jpg was being used in several language versions. Do you actually check for that, or do you simply delete? That usage in the language editions has now been lost because the duplicate was deleted without a redirect. זיו「Ziv」 • For love letters and other notes 15:41, 30 June 2026 (UTC)Reply
I found the file, restored it, and then correctly deleted it as a duplicate with a redirect, ensuring that the replacement also took place across the language versions. זיו「Ziv」 • For love letters and other notes 15:55, 30 June 2026 (UTC)Reply
@Ziv: Honestly, I don't have any preference, as long as project links aren't broken, besides just trying to do whatever the Commons community is asking of me. I once performed some file moves myself without leaving redirects, and as accused of vandalism. This is part of the problem with operating a bot on Commons. Even if I just try to do as instructed, there is not a consensus practice by all Commons editors. For most of the ones that were deleted, they were never in use yet, and the IDs had changed, so there is nothing to fix or improve about the deleted files. If one of the current file names was deleted, that should get caught and correctly renamed (with CommonsDelinker triggered) without any need for admin intervention when I re-scan, as long as there is only one version now. The Reagan photo you flagged is still confusing me; due to the deleted revisions, it is hard to reconstruct what the bot tried to tag and why, but I want to make sure there wasn't any problem on my side. Dominic (talk) 19:36, 30 June 2026 (UTC)Reply
@Dominic, the bot didn't go wrong with this Reagan file. This is actually a very special case. A file that was originally uploaded by @RandomUserGuy1738: back then and subsequently edited and renamed by DPLA bot. Then, on June 29, user @PublicDomainFan08: uploaded File:South Korean President Chun Doo Hwan meets U.S. President Ronald Reagan in the Oval Office (C439-13).jpg, a new version of it. One with more pixels and a larger file size (4.400 × 2.960, 4,26 MB, DPLA version only has 4.000 × 2.691, 3,89 MB). Subsequently, he tagged the old upload as a duplicate. The one that was first simply deleted by Yann, restored by me, and then deleted again creating a redirect. Funny, DPLA bot then recreated the file over my redirect. So, what does that mean? If someone uploads a better version of a DPLA file, there’s no point in marking it as a duplicate, because the bot will just upload it again anyway. Best regards, זיו「Ziv」 • For love letters and other notes 12:26, 1 July 2026 (UTC)Reply
@Ziv: Yeah, I see now. The logic can only go so far. The bot can't decide which one is "better", only whether it matches the official version from the institution's catalog. Whether RandomUserGuy1738 uploaded a slightly different file due to it being from a different source or because they made improvements on the media file, the point is that it has a different file hash and a different name now, and so the bot can't "see" it. By the way, I have added another number of duplicates, about a thousand. This was a controlled, discrete batch, not ongoing, and won't be added to until they drain out of the category again. Dominic (talk) 00:07, 2 July 2026 (UTC)Reply

Files that now use the {{DPLA metadata}} template.

[edit]
Latest comment: 4 days ago15 comments4 people in discussion

Good morning @Dominic: .

While deleting duplicates, I noticed that some files now have the template {{DPLA metadata}} instead of a complete file description. This made me wonder what happens if a user decides to crop the image and upload it as a new version, perhaps to remove a distracting frame or to crop a single person. And that's exactly what I did with File:St. John high school class, St. John, Washington, 1952 - DPLA - ed7c1321cfcf84324824e25c7a66c17c.jpg and reduced the image to three students. File:St. John high school class, St. John, Washington, 1952 - DPLA - ed7c1321cfcf84324824e25c7a66c17c (cropped).jpg. As I correctly suspected, the file description and license were not transferred to the cropped version; this is completely empty. This is not intended and can lead to problems. Best regards, זיו「Ziv」 • For love letters and other notes 06:50, 21 June 2026 (UTC)Reply

I forgot to mention that I had already cropped images from the DPLA bot before. Example File:Steinitz, William-4 - DPLA - 9eeab9ac335135c24b69ab7e36fc5354 (cropped).jpg, everything worked perfectly here, as the original file uses the {{Artwork}} template. זיו「Ziv」 • For love letters and other notes 07:29, 21 June 2026 (UTC)Reply
Further update: As expected, the file has now been tagged by @AntiCompositeBot: because a license is missing. I understand the idea behind {{DPLA metadata}}, it makes your files less maintenance-intensive, but if the template for derived works doesn't work properly, then it's completely unsuitable. I'd rather have more edits from DPLAbot, but users still have the option to create derived works without having to search high and low, possibly having to manually add their own templates and licenses. זיו「Ziv」 • For love letters and other notes 08:13, 21 June 2026 (UTC)Reply
Awesome! Thank you so much.
Understood. This is definitely an unintended situation. It's not an error by the bot, but certainly an undesireable interaction with the CropTool. We didn't invent the idea of template metadata pulled from SDC. For example, User:GeographBot has already been doing it for far longer. I tested how those are handled here, and it also loses most of its SDC when run through CropBot, the main difference being that its copyright license seems to be hardcoded so it doesn't get flagged by AntiCompositeBot. I think there is a solution here, but I need to work through the Lua. For example, if File:A is extracted from File:B, is it possible for Module:DPLA to fallback on File:A to File:B's SDC on any fields where they are blank? That would be the solution that works for both the bot workflow and the users using CropTool. Dominic (talk) 20:49, 21 June 2026 (UTC)Reply
@Ziv: I was able to get it to work as I was hoping, via Lua. You can see on your test file. No further edits were made to that file's SDC, but it is emitting the SDC from the {{Extracted from}} file as a fallback (until any metadata is added directly on the file, which would take precedence). This means all CropTool files that copy the template will work natively, as long as the linkage through {{Extracted from}} is kept in place. Dominic (talk) 01:11, 22 June 2026 (UTC)Reply
Dominic! You did it exactly the way I imagined and was hoping for. Yes, I'm familiar with GeographBot. I also once cropped a file uploaded by that bot. I had the same result as you. However, I then went and added the missing information myself. Perhaps your solution would also be something for @Multichill: , who runs the GeographBot? Anyway. Thank you so much for fixing this. Great job! Best regards, זיו「Ziv」 • For love letters and other notes 08:05, 22 June 2026 (UTC)Reply
Maybe on the extracted file consider a "subst" so that the wikitext gets populated, instead of relying on a different file's SDC? - Jmabel ! talk
@Jmabel: The Lua module can only affect the template display, but not make an actual edit. The issue here is that these are uploads being made by CropTool (via user accounts), not DPLA bot. DPLA bot has no knowledge of these files and isn’t maintaining them, since it also doesn’t really k ow what the user has cropped the images to and if it would be accurate to continue syncing the original metadata to it. The template is designed so that using wiki text parameters will override the SDC fallback. But actually hardcoding it by bot would need to be done on the CropTool side. Or we could run a periodic scan for these, but that is a lot of architecture and still leaves them in the state they are in now in the interim anyway. Dominic (talk) 19:48, 22 June 2026 (UTC)Reply
CropTool has been under continuous active development lately, so it's plausible that side could be addressed. Doc James, any thoughts? - Jmabel ! talk 23:37, 22 June 2026 (UTC)Reply
Sure will ask User:Bawolff to add support for this work flow if possible. Doc James (talk · contribs · email) 00:57, 23 June 2026 (UTC)Reply
@Doc James: I think the answer might be in the "Metadata to copy" feature that CropTool already has. Can all the DPLA-sourced SDC (we have a standard reference statement for all) be available options for the user to copy in there? Or, if those are pre-programmed, you can see the properties we add here: COM:DPLA/MOD. Dominic (talk) 16:15, 29 June 2026 (UTC)Reply
@Dominic: This doesn't seem to be working again File:St. John high school class, St. John, Washington, 1952 - DPLA - ed7c1321cfcf84324824e25c7a66c17c (cropped).jpg. The DPLA description is missing once more, and AntiCompositeBot has tagged it again. Why was this changed back when the problem seemed to have been solved? זיו「Ziv」 • For love letters and other notes 12:20, 5 July 2026 (UTC)Reply
I’ve identified the problem: the DPLA bot has since moved the file from which the crop was created to a new name. The issue, however, is that the cropped version isn't being updated accordingly, which in turn has resulted in the license information being missing. Ultimately, this could result in crops that were created years ago—and held a valid license—suddenly losing that license simply because the DPLA decides to move file in 2032?? to a new ID. Solution: The bot should recognize that a crop exists and then adjust it accordingly to the new ID. Best regards, זיו「Ziv」 • For love letters and other notes 12:32, 5 July 2026 (UTC)Reply
@Ziv: As long as the extracted-from file is still a redirect, seems like it would work in Lua. Let me look into it. Dominic (talk) 14:52, 5 July 2026 (UTC)Reply
@Dominic: Yes, but only because I changed it manually to the new fielname. However, this shouldn't be imposed on the average user, who is now wondering how to solve the problem and whether they did something wrong. זיו「Ziv」 • For love letters and other notes 15:35, 5 July 2026 (UTC)Reply
Addendum: The bot overwrite the old redirect for the new file. זיו「Ziv」 • For love letters and other notes 15:36, 5 July 2026 (UTC)Reply

Please tag fewer files as duplicates in batches

[edit]
Latest comment: 3 days ago6 comments3 people in discussion

Hello @Dominic

I myself wrote that I can manage several hundred per day, that certainly includes administrator @Túrelio: , who processes the category correctly, who works a lot in this category and always creates a redirect, unlike other adminstrators, who simply deletes images without creating a redirect; I personally don't think that's right, and it certainly shouldn't be the goal to simply just delete files. However, it is unacceptable for other bots, such as the one run by @Don-vip: , to stop tagging duplicates because the limit of 190 has been exceeded. Yet those duplicates ought to be processed at some point, which becomes impossible when the DPLA-bot tags several thousand files at once. However, you mustn't forget that it's not just bots using this category, but also ordinary users who pay no regard to processing limits. As far as I'm concerned, the bot can be programmed to detect when a limit has been fallen below and then tag new duplicates, like OptimusPrimeBot does. Best regards, זיו「Ziv」 • For love letters and other notes 11:47, 5 July 2026 (UTC)Reply

Another solution would, of course, be for OptimusPrimeBot to stop adhering to a limit as well, or for that limit to be raised from 190 to ???. זיו「Ziv」 • For love letters and other notes 11:57, 5 July 2026 (UTC)Reply
Thanks, I understood the need to not overload the admins maintaining the category, but didn't take into consideration that if the bots have different thresholds, DPLA_bot could crowd out others. And I also manually tagged more recently when you were active jut because I was trying to get through my backlog. Part of the problem is when the duplicates number in the thousands, I also get anxiety that another Commons editor will come across it in the wild and complain about it. I am trying to conceptualize a new approach for this where I can fully tag the backlog, but they are only released to Category:Duplicate a few at a time. Here is what I am working on: I made {{DPLA duplicate}} and Category:DPLA duplicates for deletion. The purpose is to allow my bot to fully tag the backlog, without crowding Category:Duplicate at all. Then I have a bot job that checks Category:Duplicate on a regular cadence, and only fills it up to 100 DPLA files, refilling from the backlog (using a timestamp-based moving window). This makes it a backlog that the bot can fully self-maintain without other inputs, aside from the one-time initial tag run. Dominic (talk) 18:09, 5 July 2026 (UTC)Reply
It's nice, thanks a lot @Dominic <3 vip (talk) 21:06, 5 July 2026 (UTC)Reply
Very well done @Dominic. Thanks a lot. Best regards, זיו「Ziv」 • For love letters and other notes 07:50, 6 July 2026 (UTC)Reply
Okay, great! I set the threshold at 100, and the check runs every hour. So once per hour, if the bot detects fewer than 100 DPLA files in Category:Duplicate, it fills it back up to 100. I figure since it is filling it back up to the limit every time, if I use 190, then it still crowds out the other bots, because it will use up the 190 constantly. That way the DPLA files can churn and be taken care of but without ever overfilling the backlog. Of course, you won't see it trigger until the current backlog drains back down to that level, but with thresholds set it means DPLA will back off let you work through those. Category:DPLA duplicates for deletion is up to 3000, but I haven't added the main queue I have there yet, until we give it some time to see it working to everyone's satisfaction. Dominic (talk) 16:01, 6 July 2026 (UTC)Reply

A glitch

[edit]
Latest comment: 1 hour ago3 comments2 people in discussion

Looks like something went quite wrong with your recent bot edit at File:Seattle waterfront at Pier 4, ca. 1910 - DPLA - 885aa5020b31023925ce94b55e801b8d.jpg, including loss of ImageNotes. - Jmabel ! talk 04:38, 10 July 2026 (UTC)Reply

In that file, and also in File:Tidelands SE from Occidental Hotel, ca. 1884 - DPLA - dcdd800c897fb9d1de321c352ae79db9.jpg (no ImageNotes there), my notes in the description got completely dropped. In both cases, my notes in the description are more or less incorporated by SPL's updates, so that is not disastrous, but I presume that the bot dropped the content without knowing that.

FWIW, normally when I see an edit summary like "Update description after title drift correction (DPLA ID dcdd800c897fb9d1de321c352ae79db9)" from a bot, I don't check the edit, because I assume it is just a bot doing something routine. Also, I don't necessarily watchlist every file I touch. Consequently, I might have failed to notice any number of similar errors by the bot. I'm definitely concerned that it may have killed ImageNotes on many other files, and somewhat concerned that in discarding my notes in the description for some files, significant content might have been lost. - Jmabel ! talk 05:05, 10 July 2026 (UTC)Reply

I’ll definitely look more closely into this. I can already see the cause, since this looks like it’s after a rename. The “after title drift” is a different code path than the regular one, that is invoked when the file has to be renamed too (i.e. in the years since upload the name or ID changed). It’s different because in those situations we don’t preserve all the original data (e.g. the old DPLA ID and URL are wrong and need to be overwritten). It looks like the community metadata preservation code I built didn’t touch this flow. Luckily, everything is recoverable from the edit histories, and I’ll make sure to repair these. Dominic (talk) 11:04, 10 July 2026 (UTC)Reply
Retrieved from "https://commons.wikimedia.org/w/index.php?title=User_talk:DPLA_bot&oldid=1245994821"