this post was submitted on 24 Oct 2023
265 points (91.8% liked)

Technology

34838 readers
15 users here now

This is the official technology community of Lemmy.ml for all news related to creation and use of technology, and to facilitate civil, meaningful discussion around it.

Ask in DM before posting product reviews or ads. All such posts otherwise are subject to removal.

Rules:

1: All Lemmy rules apply

2: Do not post low effort posts

3: NEVER post naziped*gore stuff

4: Always post article URLs or their archived version URLs as sources, NOT screenshots. Help the blind users.

5: personal rants of Big Tech CEOs like Elon Musk are unwelcome (does not include posts about their companies affecting wide range of people)

6: no advertisement posts unless verified as legitimate and non-exploitative/non-consumerist

7: crypto related posts, unless essential, are disallowed

founded 5 years ago
MODERATORS
MinutePhrase@lemmy.ml
265
Nightshade - A new data poisoning tool lets artists fight back against generative AI (lemmy.world)
submitted 1 year ago* (last edited 1 year ago) by ekZepp@lemmy.world to c/technology@lemmy.ml
44 comments fedilink hide all child comments
 

The tool, called Nightshade, messes up training data in ways that could cause serious damage to image-generating AI models. Is intended as a way to fight back against AI companies that use artists’ work to train their models without the creator’s permission.

ARTICLE - Technology Review

ARTICLE - Mashable

ARTICLE - Gizmodo

The researchers tested the attack on Stable Diffusion’s latest models and on an AI model they trained themselves from scratch. When they fed Stable Diffusion just 50 poisoned images of dogs and then prompted it to create images of dogs itself, the output started looking weird—creatures with too many limbs and cartoonish faces. With 300 poisoned samples, an attacker can manipulate Stable Diffusion to generate images of dogs to look like cats.

all 47 comments
sorted by: hot top controversial new old
[–] JustEnoughDucks@feddit.nl 47 points 1 year ago (3 children)

I'm interested to know how they fool the AI while keeping it invisible to the human eye. Do they make additional layers? Do they change every nth pixel? Is every poisoning associated with another poisoned object? (Will a dog always be poisoned towards a cat?, etc...)

Interesting, but a bit hard to understand.

[–] bort@feddit.de 4 points 1 year ago (1 children)

how they fool the AI while keeping it invisible to the human eye

My guess is that AI companies will try to scrape as much as possible without a human ever looking at the data.

When poisoned data start to become enough of a problem, that humans have to look over very sample, then this would increase training cost to to a point where it's no longer worth to bother with it in the first place.

[–] JustEnoughDucks@feddit.nl 14 points 1 year ago

But that has absolutely nothing to do with how the mechanism works lol. Of course they are trying to eliminate data scraping, that is the whole controversy

[–] itsralC@lemm.ee 2 points 1 year ago (1 children)

Disappointingly, the article only says that it "changes pixels in ways imperceptible to the human eye"

[–] doctorcherry@lemmy.ml 2 points 1 year ago

I think that is a feature

[–] Starshader@lemmy.ml 31 points 1 year ago (1 children)

AI using artists work is inevitable and will be a thing. We can't fight these change, we will resist these changes but eventually the majority will accept it for convenience. That's what our society do. The only chance we get to control it, is that for every use of an artist work, a little payment is made for them. Think Spotify or stuff like that. At least until an economic revolution.

[–] shapesandstuff@feddit.de 11 points 1 year ago (2 children)

Either that, or aigen companies have to hire traning set artists or something like that. That'd be better all in all

[–] qaz@lemmy.world 8 points 1 year ago* (last edited 1 year ago)

Dedicated traning artists would be expensive. They probably would buy stock art and make deals with art platforms such as Deviantart to entice creators to allow their material to be used for training for small monetary or cosmetic rewards.

[–] chicken@lemmy.dbzer0.com 5 points 1 year ago (1 children)

I would like AI models to remain free and actually published as files instead of paywalled services

[–] shapesandstuff@feddit.de -2 points 1 year ago (1 children)

Wdym remain. All the big players cost money

[–] chicken@lemmy.dbzer0.com 7 points 1 year ago

A large portion of AI art out there is made with Stable Diffusion, which can be run locally for free, and has a robust ecosystem of hobbyist trained models, LoRAs, etc. There are also somewhat competitive freely available LLM models.

Most attacks on AI that I see function as protectionism, where the biggest companies will end up being fine, but the people trying to do their own thing are the ones to be locked out.

[–] vrighter@discuss.tchncs.de 28 points 1 year ago

these don't work for any longer than a couple of days, at most.

[–] wizardbeard@lemmy.dbzer0.com 18 points 1 year ago (1 children)

Is this not just adversarial training/generation, but instead of using it to improve the model they just allow it to mess it up? Sorry, blanking on the exact term. My understanding was that some GANs are specifically trained on stuff like this to improve their abilites to differentiate.

[–] Restaldt@lemmy.world 3 points 1 year ago* (last edited 1 year ago)

Pretty much

Its on the same path as GAN but there is no adversarial network feedback - Nothing telling the generative ai it is generating bad data

Seems like GAN without the benefits for training models (which is what they wanted it seems. To mess with the training data)

I dont see how this becomes permanent since the models are already trained. Maybe if the technique becomes easy for artists to apply to their digital works and makes it into the training data for the next models

[–] kromem@lemmy.world 17 points 1 year ago* (last edited 1 year ago)

This is one of the dumbest things I've ever seen.

Anyone who thinks this is going to work doesn't understand the concept of signal to noise.

Let's say you are an artist who draws cats. And you are super worried big tech is going to be able to use your images to teach AI what a cat looks like. So you instead use this to pixel mangle it to bias towards looking like a lizard.

Over there is another artist who also draws cats and is worried about AI. So they use this tool to make cats bias towards looking like horses.

All that bias data taken across thousands of pictures of cats ends up becoming indistinguishable from noise. There's no more hidden bias signal.

The only way this would work is if the majority of all images in the training data of object A all had hidden bias towards object B (as were the very artificial conditions used in the paper).

This compounds by multiple axes for what you'd want to bias. If you draw fantasy cats, are you only biasing away from cats to dogs? Or are you also going to try to bias against fantasy to pointillism? You can always bias towards pointillism dogs, but now your poisoning is less effective combined with a cubist cat artist biasing towards anime dogs.

As you dilute the bias data by trying to cover multiple aspects that can be learned from your images by AI, you further plummet the signal into noise such that even if there was collective agreement on how to bias each individual axis, it'd be effectively worthless in a large and diverse training set.

This is dumb.

[–] qaz@lemmy.world 13 points 1 year ago* (last edited 1 year ago) (2 children)

Can you explain what the chart means? It seems like it’s supposed to show that it will degrade the output of the models when the number of poisoned samples increases, however it shows a different subject above than below. Does it morph the subject into another concept?

[–] ekZepp@lemmy.world 5 points 1 year ago (1 children)

👆 updated

[–] WhatAmLemmy@lemmy.world 16 points 1 year ago (3 children)

The problem is that the chart is shit. There's a prompt on the top and then text on the bottom that looks identical to the prompt, but is actually just what the top prompt was poisoned to look like after 100 or 300 samples.

If users have to read a paragraph of text to understand a chart, the chart is shit.

[–] bruce965@lemmy.ml 18 points 1 year ago (1 children)

A less salty way to put it would be that the chart is missing two labels: "Original prompt" and "Poisoned prompt".

[–] WhatAmLemmy@lemmy.world 4 points 1 year ago

The second isn't even a prompt. I can't fault you for getting it wrong though, because the chart is so shit!

[–] ekZepp@lemmy.world 8 points 1 year ago

Not very clear indeed. Each column is a determinate image who is been poisoned and as the lvl of poisoning increase the generated images degrade and turn in something completely different.

[–] SaltyIceteaMaker@lemmy.ml -2 points 1 year ago

Im just gonna be direct. If you cannot understand that chart you severely lack understanding of context.

If you just look at 3 pictures in one row and read the text you should easily be able to understand what the chart is about... That's like 10 year old logical thinking, if not even younger.

[–] Gabu@lemmy.world 13 points 1 year ago (1 children)

Wanna bet this can be undone in 2 seconds by running an automatic script with basic image manipulation?

AI is here to stay – sure, it sucks to get plagiarized, but there are things artists can do which AI isn't yet good at. Focus on that, instead of wasting time and energy on paliative solutions.

[–] AphoticDev@lemmy.dbzer0.com 2 points 1 year ago

The last time this popped up was months ago on reddit, and the tool they came up with did something that could be reversed as a batch job using any image manipulator. Which means somebody will write a Stable Diffusion plug-in to fix these images.

[–] iamtrashman1312@lemmy.world 6 points 1 year ago

Unrelated to much in the way of the article, but the middle tier of anime poisoned with cubism actually looks sorta cool

[–] mindbleach@sh.itjust.works 5 points 1 year ago (1 children)

This fight is over.

[–] AphoticDev@lemmy.dbzer0.com 2 points 1 year ago (1 children)

If this is all artists brought to the table, it wasn't even a fight. SD is trained on vast data sets, this little effort won't be but a drop in the ocean.

[–] mindbleach@sh.itjust.works 4 points 1 year ago

More than that - there is no need for new inputs. Massive datasets exist independently. I've got one just from a long-term habit of saving images. And my big fat pile of JPGs doesn't matter, because these models are already out there, in the wild, with communities built on screwing around with them.

The horse left the barn a year ago. It is already too late to stop this. We can bicker about moral and legal rights surrounding published content, but any suggestion of un-inventing this technology is a misguided fantasy.

There is no "if." This fight is over.

[–] lvxferre@lemmy.ml 5 points 1 year ago

The idea has some merit but it's harder to implement than it looks like. Model-based image generation is heavily biased towards typical values, so you'd need a lot of poison to do it. And that poison would need to be consistent - it doesn't work if you tell the model now that cats are dogs and then that ferrets are dogs, you need to pick one.

I'm rather entertained by the amount of fallacies and assumptions ITT though. I get that you guys are excited with model-based image gen; frankly, I'm the same when it comes to text gen. But those two things won't help, learn the difference between "X is true" and "I want X to be true".

[–] Zekas@lemmy.world 4 points 1 year ago (1 children)

Wasn't there already a tool like this called Glaze?

[–] drdiddlybadger@pawb.social 8 points 1 year ago

This is pretty much Glaze 2. It just intentionally poisons the data set with specific targets so model is more fucked. Originally it was just noise being put in and ultimately a image that had been glazed would just get tossed. With this, the image will actually fuck up the resulting model of there is enough poisoned data included.

Probably, I'm not an expert obviously.

[–] be_excellent_to_each_other@kbin.social 4 points 1 year ago

Begun, the AI wars have.

[–] TheFriar@lemm.ee 3 points 1 year ago (1 children)

I absolutely love this. I’m not even an artist, but I’m giddy over this.

[–] AphoticDev@lemmy.dbzer0.com 1 points 1 year ago

Don't be too gidy, it won't work. SD is already trained on poisoned datasets to help it differentiate poorly generated images. We call it "adversarial training". If this was gonna stop us from making AI artwork, , it already would have.

[–] Zerush@lemmy.ml 1 points 1 year ago* (last edited 1 year ago)

Meanwhile with AI videos (more in the Aibient channel). Intoxicate AI LOL, it made it as a feature

https://piped.kavin.rocks/watch?v=g8yX1Jfq8K4

[–] sunbeam60@lemmy.one -2 points 1 year ago (1 children)

The only solution, if there is one, is to put your art on the blockchain and specifically license against it being used without attribution on same blockchain and the find some kind of license model that trickles value up the chain.

Even that won’t work, I suspect.

[–] Saltblue@lemmy.world 11 points 1 year ago (1 children)

put your art on the blockchain

Stopped reading

[–] sunbeam60@lemmy.one 3 points 1 year ago

Ha ha me too and I wrote it.

I’m very aware that there’s nothing to stop a bad actor from ignoring whatever is on the blockchain. But imagine removing all the web3/cryptobro bullshit that makes us all sick and instead just look at it as a record of who’s done what to which file. It could also be a centralised DB but it seems no one should have that power. A smart contract (aka ethereum) that says “anything derived from this sends some transactional fee up toward the originator”.

I mean I’m aware it won’t work.

I’m just saying that I can’t come up with anything better and so I also believe the battle is lost.