Design of Experiments 101: Cross Validation

What is an experiment?

An experiment is a procedure that you perform in order to validate (or to reject) your hypothesis.

Your hypothesis might be that the selection strategy, the classifier (regressor), or a smart combination of those that you developed performs better than others. Or maybe you just want to let your approaches in the wild (on your data) and assess the results.

For the sake of simplicity, let’s assume that you have a paradigm H (your hypothesis), a data set X, and a performance measure E (this is how you assess the performance of your approach numerically; e.g. classification accuracy).

The following approach works for supervised learning too, not just for active learning.

A simple example

The main idea behind design of experiments is:

the design of the experiment is similar to a contest.

The Contest: Alice has a dataset consisting of 100 data points and wants to know if Bob or Carl is the better data scientist. So, she gives Bob and Carl 75 data points and asks each of them to provide the best model they can achieve. After that Alice will compare both models on the 25 data points, she held back.

The Optimization: Now, both data scientists try to find the best parameters for their model. They also split the data: 60 for training and 15 for validation. After training several models with different parameters on the 60 data points, each of them chooses the model which performed best on the remaining 15 data points.

The Comparison: Finally, Alice will evaluate the final models of both data scientists on her held out data. Bob wins if his model performs best and Carl respectively.

Our terminology

In the following, we use these terms to describe the different kinds of subsets (see also wikipedia):

  • Outer training set: the data Bob and Carl are given by Alice to find their best approach (75 data points)
  • Outer test set (often: test or evaluation set): the data Alice held back to test Bob’s and Carl’s approach (25 data points)
  • Inner training set (often: training set): the data Bob and Carl used to train a model with specific parameters of their approach (60 data points)
  • Inner test set (often: validation set): the data Bob and Carl used to determine the best parameter set (15 data points)

How can Bob and Carl do better (improve the generalization of their training procedure)?

So far, both data scientist just had one fixed training set (inner training set) and one validation set (inner test set). By random it could happen that one test set is particularly difficult for parameter setting and easy for another. Hence, we should ensure that every instance has been used for testing.

In k-fold cross validation, the data given by Alice (75 data points) is split in \(k=5\) folds. Hence, they have 5 subsets with 15 instances each. To predict the labels of the first fold, the data from folds 2, 3, 4, 5 is used for training. For the second fold, the algorithm is trained on folds 1, 3, 4, 5, etc. This methodology is much more robust and therefore leads to better results. Hence, it is more probable that the parameter setting which performed best actually is the best for the given data.

But now, one problem occurs. For the best parameter setting, each data scientist has 5 different model because of the k-fold cross validation. As Carl did not know what to do, he chose one by random. Bob had a better idea: He used the parameter setting, he found out was best, and trained the model on all data that he was given.

How can Alice do better?

Alice is faced with a similar situation as Bob and Carl. Maybe, someone just got lucky or the selection of training resp. test instances has been better for one of the competitors. Hence, Alice also performs k-fold cross validation (here \(k=4\)). Hence, Bob and Carl are asked to provide 4 different models and Alice checks if the results are consistent.

To be even more certain, she calculates only one performance value for one k-fold cross validation. Then she repeats the selection of instances multiple times to be certain that the results are not random.

Summary: How do you split your data?

The main idea of cross validation is to prevent that the model had seen the test data during training. This means that test data has neither been used for training or tuning. If we want to rank different algorithms with their best parameter setting, we need the two-staged cross validation. Hence, algorithms selection is the outer cross validation and on each training set, we perform a separate inner cross validation. More details can be found in the wikipedia pages mentioned above.

If you are interested how to evaluate active learning algorithms, please see the paper:
Challenges of Reliable, Realistic and Comparable Active Learning Evaluation by Kottke, Calma et al.

17 Replies to “Design of Experiments 101: Cross Validation”

  1. Hi there, I log on to your blogs like every week. Your
    story-telling style is witty, keep doing what you’re doing!

    Also visit my blog post; 구글상위노출교육

  2. I think that is among the such a lot important information for me.
    And i’m happy studying your article. But want to statement on few common issues,
    The site taste is great, the articles is actually excellent
    : D. Excellent job, cheers

  3. Excellent post. I was checking constantly this weblog and I am inspired!
    Very useful information specially the ultimate phase :
    ) I care for such information a lot. I used to be looking for this particular info for a long time.
    Thank you and good luck.

  4. cc dumps 2020 Good validity rate Purchasing Make good job for MMO Pay all site activate your card now
    for worldwide transactions.
    ————-CONTACT———————–
    WEBSITE : >>>>>> Cvvdumps✷ Site

    —– HERE COMES THE PRICE LIST ———–
    ***** CCV US:
    – US MASTER CARD = $2,8 per 1 (buy >5 with price $3 per 1).

    – US VISA CARD = $2,7 per 1 (buy >5 with price $2.5
    per 1).
    – US AMEX CARD = $4,1 per 1 (buy >5 with price $2.5 per 1).

    – US DISCOVER CARD = $2,2 per 1 (buy >5 with price
    $3.5 per 1).
    – US CARD WITH DOB = $15 per 1 (buy >5 with
    price $12 per 1).
    – US FULLZ INFO = $40 per 1 (buy >10 with price $30 per 1).

    ***** CCV UK:
    – UK CARD NORMAL = $2,7 per 1 (buy >5 with price $3 per 1).

    – UK MASTER CARD = $2,9 per 1 (buy >5 with price $2.5 per 1).

    – UK VISA CARD = $2,8 per 1 (buy >5 with price $2.5 per 1).

    – UK AMEX CARD = $4,5 per 1 (buy >5 with price $4 per 1).

    $5,9

    – UK CARD WITH DOB = $15 per 1 (buy >5 with price
    $14 per 1).
    – UK WITH BIN = $10 per 1 (buy >5 with price $9 per 1).
    – UK WITH BIN WITH DOB = $25 per 1 (buy >20 with price $22 per 1).

    – UK FULLZ INFO = $40 per 1 (buy >10 with price $35 per 1).

    ***** CCV AU:
    – AU MASTER CARD = $5.5 per 1 (buy >5 with price $5 per 1).

    – AU VISA CARD = $5.5 per 1 (buy >5 with price $5 per 1).

    – AU AMEX CARD = $8.5 per 1 (buy >5 with price $8 per 1).

    – AU DISCOVER CARD = $8.5 per 1 (buy >5 with price $8
    per 1).
    ***** CCV CA:
    – CA MASTER CARD = $6 per 1 (buy >5 with price $5 per 1).

    – CA VISA CARD = $6 per 1 (buy >5 with price $5 per 1).

    – CA VISA BUSINESS = $14 per 1 (buy >5 with price $13 per 1).

  5. cc dumps free Good validity rate Purchasing Make good job for you Pay on web activate your card now for worldwide transactions.

    ————-CONTACT———————–
    WEBSITE : >>>>>> Cvvdumps✷ Site

    —– HERE COMES THE PRICE LIST ———–
    ***** CCV US:
    – US MASTER CARD = $2,5 per 1 (buy >5 with price $3 per 1).

    – US VISA CARD = $3 per 1 (buy >5 with price $2.5 per 1).

    – US AMEX CARD = $2,2 per 1 (buy >5 with price $2.5 per 1).

    – US DISCOVER CARD = $2,9 per 1 (buy >5 with price $3.5 per 1).

    – US CARD WITH DOB = $15 per 1 (buy >5 with price $12 per 1).

    – US FULLZ INFO = $40 per 1 (buy >10 with price $30 per 1).

    ***** CCV UK:
    – UK CARD NORMAL = $2,5 per 1 (buy >5 with price $3 per 1).

    – UK MASTER CARD = $2,7 per 1 (buy >5 with price $2.5 per 1).

    – UK VISA CARD = $3 per 1 (buy >5 with price $2.5 per 1).

    – UK AMEX CARD = $4,2 per 1 (buy >5 with price $4 per 1).

    $

    – UK CARD WITH DOB = $15 per 1 (buy >5 with price $14 per 1).

    – UK WITH BIN = $10 per 1 (buy >5 with price $9 per 1).
    – UK WITH BIN WITH DOB = $25 per 1 (buy >20 with price $22 per
    1).
    – UK FULLZ INFO = $40 per 1 (buy >10 with price $35 per 1).

    ***** CCV AU:
    – AU MASTER CARD = $5.5 per 1 (buy >5 with price $5 per 1).

    – AU VISA CARD = $5.5 per 1 (buy >5 with price $5 per
    1).
    – AU AMEX CARD = $8.5 per 1 (buy >5 with price $8
    per 1).
    – AU DISCOVER CARD = $8.5 per 1 (buy >5 with price $8 per 1).

    ***** CCV CA:
    – CA MASTER CARD = $6 per 1 (buy >5 with price $5 per 1).

    – CA VISA CARD = $6 per 1 (buy >5 with price $5 per 1).

    – CA VISA BUSINESS = $14 per 1 (buy >5 with price $13 per 1).

  6. You are my breathing in, I have few web logs and occasionally run out from brand :
    ).

  7. I really value your piece of work, Great post.

  8. Good way of telling, and fastidious paragraph to take facts regarding my presentation topic, which i am going to convey in college.

  9. Hi i am kavin, its my first occasion to commenting anywhere, when i read this article i
    thought i could also make comment due to this good article.

  10. You are a very clever person!

  11. This site certainly has all the information I wanted concerning this subject and
    didn’t know who to ask.

  12. ว้าว ฉันบังเอิญไปเจอบทความนี้ใน Google ฉันรู้สึกสนใจและแท็กมัน แน่นอนว่าฉันชอบเว็บไซต์ของคุณมากเช่นกัน มันกระชับมากและมีบทความและข้อมูลที่น่าสนใจมากมาย fun88 ถอนเงิน

  13. 환영합니다.
    when i read this paragraph i thought i could
    also create comment due to this brilliant paragraph.
    새로운 에볼루션바카라 사이트도 방문해보세요
    새로운 온라인카지노 경험을 하실 수 있습니다.

  14. Fun88 เป็นเว็บไซต์การพนันออนไลน์ชั้นนำของเอเชียที่นำเสนอเกมที่หลากหลาย รวมถึงการพนันกีฬา คาสิโนออนไลน์ ลอตเตอรี่ สล็อต และอื่นๆ อีกมากมาย เมื่อเข้าสู่ระบบ Fun88 ผู้เล่นจะเข้าสู่แพลตฟอร์มที่ทันสมัยพร้อมผลิตภัณฑ์ที่หลากหลายให้เลือก เว็บไซต์เสนอราคาเดิมพันที่ดีที่สุดพร้อมทั้งโปรโมชั่นที่น่าสนใจมากมาย
    ทางเข้าล่าสุด:เว บ พน น fun88

  15. Fun88 เป็นเว็บไซต์ที่น่าสนใจสำหรับผู้ที่หลงใหลในการเล่นการพนันออนไลน์ ทางเข้าที่เร็วและปลอดภัยของเว็บไซต์นี้จะทำให้คุณสามารถเข้าถึงการเดิมพันบอลและคาสิโนสดได้อย่างรวดเร็วและปลอดภัยทุกวัน.
    แจก รหัส คูปอง โบนัส fun88 มีทางเข้าที่รวดเร็วที่สุด คุณไม่ต้องเสียเวลาในการรอหรือโหลดหน้าเว็บนานเนื่องจากความเร็วของระบบที่พร้อมให้บริการตลอดเวลา ทำให้คุณสามารถเพลิดเพลินกับการเดิมพันที่คุณชื่นชอบได้อย่างต่อเนื่อง.
    แทงหวยเว บ fun88

  16. ขอบคุณสำหรับการแบ่งปันและยินดีต้อนรับสู่ Fun88 แพลตฟอร์มลอตเตอรีสมัยใหม่ที่ผู้เล่นสามารถเพลิดเพลินกับประสบการณ์ผลิตภัณฑ์ที่หลากหลายและหลากหลาย
    ดาวน์โหลด fun88

  17. Trong lĩnh vực cá cược bóng đá, ty le keo nhà cai là một yếu tố quan trọng mà mọi người chơi cần phải hiểu rõ.

Leave a Reply


CAPTCHA Image
Reload Image