How to calculate uniqueness algorithmically?

by 10 replies
12
Does anyone know how to calculate the uniqueness of an article in spinning syntax? Or at least how to determine how 'different' two articles are?

Some sample code would be great (language doesn't matter).

My goal is to create x unique articles from an article in spinning syntax that are as 'different' as possible.
#programming #algorithmically #calculate #uniqueness
  • Michael,

    You could create a hash of all words in each article, then compare two articles and score based on the percentage of words that don't appear in both articles.
    • [1] reply
    • There is an online web page that does this for you. CopyScape is the name of the website.
      plagiarismchecker is also good
      • [1] reply
  • Would something like PHP's similar_text function work for you?
  • Michael,

    I was suggesting comparing a hash of words, so that the order of words as well as the number of times each word appears, will not be relevant.

    The score will be based on the percentage of unique words that appear in one article but not the other.
    • [1] reply
    • Consider the following text in spinning syntax:
      Code:
      {This is a sentence.|A sentence is this.}
      According to your suggestion the uniqueness of this text would be 0%, wouldn't it?

      However, TheBestSpinner calculates 80%, whatever that means...
  • Hi Michael,

    Never said anything about a spinning syntax.

    Just wrote up this sample code for you:

    <?php
    $article1 = "This is a sentence.";
    $article2 = "A sentence is this";

    $hash1 = BuildWordHash($article1);
    $hash2 = BuildWordHash($article2);

    echo "The two articles are ".CompareWordHash($hash1, $hash2)."% alike\r\n";

    function BuildWordHash($body)
    {
    $words = array();

    // Get rid of garbage
    $body = str_replace(array(",",".",":",";"),array("","","", ""),$body);

    // Build hash
    $arr_words = explode(" ", $body);
    foreach ($arr_words as $word)
    $words[] = strtolower($word);

    return $words;
    }

    function CompareWordHash($arr_words1, $arr_words2)
    {
    // Initialize
    $unique_words = 0;

    // Set these for easier access
    $total_words = count(array_merge($arr_words1,$arr_words2));

    // Start from arr_words1
    foreach ($arr_words1 as $word)
    if (!in_array(strtolower($word),$arr_words2))
    {
    unset($arr_words1[$word]);
    $unique_words++;
    }


    // Now move on to arr_words2
    foreach ($arr_words2 as $word)
    if (!in_array(strtolower($word),$arr_words1))
    {
    unset($arr_words2[$word]);
    $unique_words++;
    }

    // Set these for easier access
    $percentage = number_format((1-($unique_words / $total_words))*100,0);

    return $percentage;
    }
    ?>

    Enjoy!
    • [1] reply
    • That's exactly what I said. The code calculates 0% uniqueness (or: The two articles are 100% alike), while TheBestSpinner says that the uniqueness is 80%.

      I wonder what these 80% mean...

      Edit: I just recognized that the calculation in TheBestSpinner doesn't make sense:

      Guess what the uniequeness of the following 'article' is?
      Code:
      {test test test test test test|test test test test test test}
      You were right! 86%!
  • Hi Michael,

    You're welcome :-)

    Not sure how good TheBestSpinner is, but the code I pasted works
  • I have spend a lot of time in spinning articles but i found it is just wastage of time you can write article on notepad2 it will help you a lot in writing articles.
  • Banned
    [DELETED]

Next Topics on Trending Feed