social.coop

0 posts0 participants0 posts today

Johann Joets<a href="https://www.r-bloggers.com/2025/04/setting-up-cross-validation-caret-package-in-r-a-step-by-step-guide/?utm_source=phpList&utm_medium=email&utm_campaign=R-bloggers-daily&utm_content=HTML" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://www.r-bloggers.com/2025/04/setting-up-cross-validation-caret-package-in-r-a-step-by-step-guide/?utm_source=phpList&utm_medium=email&utm_campaign=R-bloggers-daily&utm_content=HTML</a> <a href="https://mastodon.social/tags/rstats" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#rstats</a> <a href="https://mastodon.social/tags/crossValidation" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#crossValidation</a> <a href="https://mastodon.social/tags/modeling" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#modeling</a>

neuralrowNew article out folks Check it out <a href="https://medium.com/@neuralrow/cross-validation-1df416898697" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://medium.com/@neuralrow/cross-validation-1df416898697</a><a href="https://mastodon.social/tags/DataScience" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#DataScience</a> <a href="https://mastodon.social/tags/MachineLearning" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#MachineLearning</a> <a href="https://mastodon.social/tags/CrossValidation" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#CrossValidation</a> <a href="https://mastodon.social/tags/LinearRegression" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#LinearRegression</a> <a href="https://mastodon.social/tags/DeepLearning" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#DeepLearning</a> <a href="https://mastodon.social/tags/NeuralNetworks" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#NeuralNetworks</a>

nf-corePipeline release! nf-core/drugresponseeval v1.0.0 - 1.0.0!Please see the changelog: <a href="https://github.com/nf-core/drugresponseeval/releases/tag/1.0.0" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://github.com/nf-core/drugresponseeval/releases/tag/1.0.0</a><a href="https://mstdn.science/tags/celllines" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#celllines</a> <a href="https://mstdn.science/tags/crossvalidation" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#crossvalidation</a> <a href="https://mstdn.science/tags/deeplearning" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#deeplearning</a> <a href="https://mstdn.science/tags/drugresponse" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#drugresponse</a> <a href="https://mstdn.science/tags/drugresponseprediction" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#drugresponseprediction</a> <a href="https://mstdn.science/tags/drugs" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#drugs</a> <a href="https://mstdn.science/tags/fairprinciples" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#fairprinciples</a> <a href="https://mstdn.science/tags/generalization" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#generalization</a> <a href="https://mstdn.science/tags/hyperparametertuning" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#hyperparametertuning</a> <a href="https://mstdn.science/tags/machinelearning" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#machinelearning</a> <a href="https://mstdn.science/tags/randomizationtests" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#randomizationtests</a> <a href="https://mstdn.science/tags/robustnessassessment" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#robustnessassessment</a> <a href="https://mstdn.science/tags/training" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#training</a> <a href="https://mstdn.science/tags/nfcore" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#nfcore</a> <a href="https://mstdn.science/tags/openscience" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#openscience</a> <a href="https://mstdn.science/tags/nextflow" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#nextflow</a> <a href="https://mstdn.science/tags/bioinformatics" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#bioinformatics</a>

Chloé Azencott⬆️ 6) thankfully, Wager (2020) <a href="https://doi.org/10.1080/01621459.2020.1727235" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://doi.org/10.1080/01621459.2020.1727235</a> shows that cross-validation is asymptotically consistant for model selection, so while what we're doing gives us poor estimates of generalization error and bad error bars, at least it's valid for model selection.<a href="https://lipn.info/tags/machineLearning" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#machineLearning</a> <a href="https://lipn.info/tags/statistics" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#statistics</a> <a href="https://lipn.info/tags/crossValidation" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#crossValidation</a>

Chloé Azencott⬆️ 5) Bates et al. (2023) <a href="https://doi.org/10.1080/01621459.2023.2197686" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://doi.org/10.1080/01621459.2023.2197686</a> propose a nested cross-validation estimator of generalization error that's unbiased and has an unbiased mean squared error estimator. It's computationally quite intensive. I played a bit with it, and my in high-dimensional set ups (large p small n) I got error bars that had indeed good coverage of the generalization error, but were also covering most of the [0, 1] interval, which is less helpful.⬇️<a href="https://lipn.info/tags/machineLearning" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#machineLearning</a> <a href="https://lipn.info/tags/statistics" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#statistics</a> <a href="https://lipn.info/tags/crossValidation" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#crossValidation</a>

Chloé Azencott⬆️ 4) in any case, error bars are wrong, because it's impossible to get an unbiased estimator of the mean squared error of an estimator that's based on a single fold of cross-validation, as shown by Bengio & Grandvalet (2004) <a href="https://dl.acm.org/doi/10.5555/1005332.1044695" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://dl.acm.org/doi/10.5555/1005332.1044695</a>⬇️ <a href="https://lipn.info/tags/machineLearning" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#machineLearning</a> <a href="https://lipn.info/tags/statistics" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#statistics</a> <a href="https://lipn.info/tags/crossValidation" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#crossValidation</a>

Chloé Azencott⬆️ 3) cross-validation estimators are better estimators of *expected test error* (across all possible training sets) than of *generalization error* of a model. This has been known for a while and even appears in The Elements of Statistical Learning, so I should have known about this much earlier. Bates et al. (2023) <a href="https://doi.org/10.1080/01621459.2023.2197686" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://doi.org/10.1080/01621459.2023.2197686</a> show why this is for linear models.⬇️ <a href="https://lipn.info/tags/machineLearning" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#machineLearning</a> <a href="https://lipn.info/tags/statistics" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#statistics</a> <a href="https://lipn.info/tags/crossValidation" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#crossValidation</a>

Chloé Azencott⬆️ 2) (not a surprise, but worth remembering): cross-validation error bars can be very large when sample sizes are small (unsurprisingly, due to the \( \frac{1}{\sqrt{n}} \) factor). This is discussed for example regarding microarray studies in Braga-Neto & Dougherty (2004) <a href="https://doi.org/10.1093/bioinformatics/btg419" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://doi.org/10.1093/bioinformatics/btg419</a> and <a href="https://mastodon.social/@GaelVaroquaux" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@GaelVaroquaux</a> (2018) regarding brain image analysis <a href="https://doi.org/10.1016/j.neuroimage.2017.06.061" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://doi.org/10.1016/j.neuroimage.2017.06.061</a>⬇️ <a href="https://lipn.info/tags/machineLearning" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#machineLearning</a> <a href="https://lipn.info/tags/statistics" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#statistics</a> <a href="https://lipn.info/tags/crossValidation" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#crossValidation</a>

Chloé Azencott⬆️ Reading the discussion of the paper by other statisticians is enlightening as to how the tone of scientific discourse has mercifully changed in 50 years.Also, "The term 'assessment' is preferred to 'validation' which has a ring of excessive confidence about it."⬇️<a href="https://lipn.info/tags/machineLearning" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#machineLearning</a> <a href="https://lipn.info/tags/statistics" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#statistics</a> <a href="https://lipn.info/tags/crossValidation" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#crossValidation</a>

Chloé AzencottWe were discussing cross-validation estimates of model performance recently with colleagues, and I dug a bit in the literature to better understand where we're at.This is not my topic of expertise, but here are a few tidbits I'd like to share.1) cross-validation has been the topic of much discussion for many decades. Stone (1974) <a href="https://www.jstor.org/stable/2984809" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://www.jstor.org/stable/2984809</a> gives a good overview of what precedes. ⬇️ <a href="https://lipn.info/tags/machineLearning" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#machineLearning</a> <a href="https://lipn.info/tags/statistics" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#statistics</a> <a href="https://lipn.info/tags/crossValidation" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#crossValidation</a>

Nathan PavlovicEnjoying the discussion of cross-validation methods for use of sensor data for air quality applications at the EPA air sensor QA workshop. It’s easy to overestimate how well you are doing with sensor data corrections or fusion applications unless a rigorous independent test approach is used <a href="https://fediscience.org/tags/airquality" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#airquality</a> <a href="https://fediscience.org/tags/airpollution" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#airpollution</a> <a href="https://fediscience.org/tags/crossvalidation" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#crossvalidation</a> <a href="https://fediscience.org/tags/lowcostsensors" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#lowcostsensors</a> <a href="https://airpollution.science/@dwestervelt" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@dwestervelt</a> <a href="https://www.epa.gov/amtic/2023-air-sensors-quality-assurance-workshop" rel="nofollow noopener noreferrer" target="_blank">https://www.epa.gov/amtic/2023-air-sensors-quality-assurance-workshop</a>

Daniele de Rigo3/ <a href="https://hostux.social/tags/Feynman" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Feynman</a>: "it doesn’t make any sense to calculate after the event. You see, you found the peculiarity, and so you selected the peculiar case" <a href="https://archive.org/details/meaningofitallth0000feyn_d8d3/page/80/mode/2up?q=%22it+doesn%E2%80%99t+make+any+sense+to+calculate+after+the+event%22&view=theater" rel="nofollow noopener noreferrer" target="_blank">https://archive.org/details/meaningofitallth0000feyn_d8d3/page/80/mode/2up?q=%22it+doesn%E2%80%99t+make+any+sense+to+calculate+after+the+event%22&view=theater</a>Special trending case: <a href="https://hostux.social/tags/CrossValidation" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#CrossValidation</a> (where data for selecting/tuning a model are also used to test it, with allegedly "clever" methods to avoid fooling oneself) and other <a href="https://hostux.social/tags/MachineLearning" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#MachineLearning</a> math. tricks where many dimensions/parameters are tuned by using much less dataWithout a deep understanding, black-box tools lead astray

Tiago F. R. RibeiroModel Evaluation, Model Selection, and Algorithm Selection in Machine Learning<a href="https://mastodon.social/tags/MachineLearning" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#MachineLearning</a> <a href="https://mastodon.social/tags/ModelEvaluation" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ModelEvaluation</a> <a href="https://mastodon.social/tags/CrossValidation" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#CrossValidation</a> <a href="https://mastodon.social/tags/HyperparameterOptimization" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#HyperparameterOptimization</a> <a href="https://arxiv.org/pdf/1811.12808.pdf" rel="nofollow noopener noreferrer" target="_blank">https://arxiv.org/pdf/1811.12808.pdf</a>

Aki VehtariNew paper "Cross-validatory model selection for Bayesian autoregressions with exogenous regressors" with Alex Cooper, <a href="https://fosstodon.org/@dan_p_simpson" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@dan_p_simpson</a>, Lauren Kennedy, and Catherine ForbesOne FAQ is "Can you use LOO or cross-validation in general for time series?" The short answer is "Yes", and I've had a longer answer in CV-FAQ <a href="https://avehtari.github.io/modelselection/CV-FAQ.html#9_Can_cross-validation_be_used_for_time_series" rel="nofollow noopener noreferrer" target="_blank">https://avehtari.github.io/modelselection/CV-FAQ.html#9_Can_cross-validation_be_used_for_time_series</a>Now we have a better answer on what kind of cross-validation is good with timeseries!<a href="https://bayes.club/tags/PaperThread" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#PaperThread</a> <a href="https://bayes.club/tags/Bayesian" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Bayesian</a> <a href="https://bayes.club/tags/CrossValidation" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#CrossValidation</a>

Clara BurgardWe also looked at the influence of the average domain used for the input properties and we conducted a <a href="https://mastodon.green/tags/CrossValidation" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#CrossValidation</a> to assess how the parameterisations perform on time steps and ice shelves they have not seen during <a href="https://mastodon.green/tags/tuning" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#tuning</a>.

Recent searches

Search options

Administered by:

Server stats:

#crossvalidation