Before I answer the question, I have a question. Am I right in understanding this as the situation?

1. Old website is created, content is added.
2. Content is used elsewhere, perhaps in an article directory and syndicated elsewhere.
3. Old website is taken down, original content is removed from the SERPs.
4. New website is put up, content added back in.

Is that right?

You wouldn't be penalized if the old site has been taken down because they will not appear on Google's index. But if you built a new site and the old site is still indexed then it is duplicate content. Find out if your old site have been de-indexed and you need not worry about it.

If you want to double check, you can use copyscape or plagium to check for matches in all of your new website's content.

If you have control over it i would say do a 301 redirect to it

If you took the website down, but just offline, everything is still there\
if you go back up.

There's no duplicate content penalty. People confuse the same content
as duplicate content for a penalty. Every big site has the same content
all over the place. People just go gaga about duplicate content without
using their heads, or they just don't know.

If you took the website down, and I mean down and out, obliterated
all content, then the content that's indexed will slowly roll
off. Although you could still view it with the google cache.

No reason to do anything, really, except have a custom 404 error page,
with respect to any old content floating around.

Paul

I heard once that once you put someone out there in cyber space somehow it will always be out there. I know that's usually only for certain operations though, such as when you post items on a social network timeline.

As far as your own website, I do agree with the above. It usually will slowly disappear, usually within a matter of a few minutes but perhaps it could take a few days or longer. Oh, and for this I don't see why you should get penalized for duplicate content.

Duplicate content is when you have the same content on more than one page of a site.

Well, lets even assume what you have is a duplicate content on your site and you took it down rewrite/changed the content entirely for every page then you have all new content.

Changes will not be immediate but over time it will wipe off, you can speed things up by building back links to your site and getting it re-indexed.