social.coop

Catalyst CooperativeEverything is a lot right now. Want to tune it all out for ~10 minutes and help us better understand your energy data needs? We're kicking off our first (hopefully annual!) PUDL community survey. Please boost! Both current and possible future PUDL users encouraged.<a href="https://forms.gle/cT1ZY4yunhza3YLK9" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://forms.gle/cT1ZY4yunhza3YLK9</a><a href="https://mastodon.energy/tags/EnergyTransition" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#EnergyTransition</a> <a href="https://mastodon.energy/tags/PUDL" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#PUDL</a> <a href="https://mastodon.energy/tags/OpenData" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#OpenData</a> <a href="https://mastodon.energy/tags/EnergyData" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#EnergyData</a> <a href="https://mastodon.energy/tags/OpenSource" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#OpenSource</a>

Catalyst CooperativeFor the energy data nerds: we've got a new data release out. PUDL v2024.11.0 includes quarterly updates to EIA 860M, EIA 923 year-to-date, EIA 930, EPA CEMS, and final 2023 data for the EIA 861. Comment in this GitHub discussion if you find anything weird. (or just to say Hi 👋)<a href="https://github.com/orgs/catalyst-cooperative/discussions/3967" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://github.com/orgs/catalyst-cooperative/discussions/3967</a><a href="https://mastodon.energy/tags/EnergyTransition" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#EnergyTransition</a> <a href="https://mastodon.energy/tags/OpenData" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#OpenData</a> <a href="https://mastodon.energy/tags/PUDL" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#PUDL</a>

STE<a href="https://troet.cafe/tags/Fensterfreitag" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Fensterfreitag</a> <a href="https://troet.cafe/tags/Photography" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Photography</a> <a href="https://troet.cafe/tags/Germersheim" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Germersheim</a> <a href="https://troet.cafe/tags/Spiegelung" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Spiegelung</a> <a href="https://troet.cafe/tags/Pf%C3%BCtze" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Pfütze</a> <a href="https://troet.cafe/tags/mirror" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#mirror</a> <a href="https://troet.cafe/tags/pudl" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#pudl</a>

Catalyst CooperativeWe want to apply to the Google Season of Docs for <a href="https://mastodon.energy/tags/PUDL" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#PUDL</a> but have never worked with an outside technical writer before. Does anybody have someone to recommend? It's a <a href="https://mastodon.energy/tags/Python" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Python</a> project focused on producing open data describing the US energy system.Cc: <a href="https://fosstodon.org/@turingway" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@turingway</a> <a href="https://hachyderm.io/@choldgraf" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@choldgraf</a> <a href="https://fosstodon.org/@yabellini" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@yabellini</a> <a href="https://fosstodon.org/@leahawasser" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@leahawasser</a><a href="https://mastodon.energy/tags/PyData" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#PyData</a> <a href="https://mastodon.energy/tags/WriteTheDocs" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#WriteTheDocs</a> <a href="https://mastodon.energy/tags/EnergyTransition" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#EnergyTransition</a> <a href="https://mastodon.energy/tags/OpenData" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#OpenData</a> <a href="https://mastodon.energy/tags/OpenSource" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#OpenSource</a> <a href="https://mastodon.energy/tags/EnergyMastodon" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#EnergyMastodon</a>

Tod Robbins<a href="https://mastodon.energy/@catalystcoop" class="u-url mention">@catalystcoop</a> <a href="https://social.coop/@ZaneSelvans" class="u-url mention">@ZaneSelvans</a> are there any other open utility databases/projects besides <a href="https://social.coop/tags/PUDL" class="mention hashtag" rel="tag">#PUDL</a>?

Zane SelvansNow that we're putting all our denormalized output tables and analyses into the <a href="https://social.coop/tags/PUDL" class="mention hashtag" rel="tag">#PUDL</a> DB, we've got a lot more <a href="https://social.coop/tags/metadata" class="mention hashtag" rel="tag">#metadata</a> to manage, and are trying to figure out how to best combine existing tools to do it.GitHub Discussion: <a href="https://github.com/orgs/catalyst-cooperative/discussions/2546" target="_blank" rel="nofollow noopener noreferrer" translate="no">https://github.com/orgs/catalyst-cooperative/discussions/2546</a>Currently we store column, table, and dataset level information in big JSON-ish <a href="https://social.coop/tags/python" class="mention hashtag" rel="tag">#python</a> data structures, which are converted into objects using <a href="https://fosstodon.org/@pydantic" class="u-url mention">@pydantic</a> models based (loosely) on the <a href="https://social.coop/tags/FrictionlessData" class="mention hashtag" rel="tag">#FrictionlessData</a> tabular data package abstractions.<a href="https://social.coop/tags/datadon" class="mention hashtag" rel="tag">#datadon</a>

Zane SelvansThe <a href="https://data-folks.masto.host/@dagster" class="u-url mention">@dagster</a> folks interviewed us and did a write-up of our migration of <a href="https://social.coop/tags/PUDL" class="mention hashtag" rel="tag">#PUDL</a> from a messy DIY <a href="https://social.coop/tags/Python" class="mention hashtag" rel="tag">#Python</a> ETL to using their orchestration framework, which has thus far been a very positive experience. Unlike most of their users we are producing <a href="https://social.coop/tags/OpenData" class="mention hashtag" rel="tag">#OpenData</a> outputs. Very curious to see if other non-profit / open-data users will adopt the platform:<a href="https://dagster.io/blog/catalyst-cooperative-case-study" target="_blank" rel="nofollow noopener noreferrer" translate="no">https://dagster.io/blog/catalyst-cooperative-case-study</a><a href="https://social.coop/tags/DataEngineering" class="mention hashtag" rel="tag">#DataEngineering</a> <a href="https://social.coop/tags/datadon" class="mention hashtag" rel="tag">#datadon</a> <a href="https://social.coop/tags/EnergyMastodon" class="mention hashtag" rel="tag">#EnergyMastodon</a> <a href="https://social.coop/tags/OpenSource" class="mention hashtag" rel="tag">#OpenSource</a> <a href="https://social.coop/tags/EnergyTransition" class="mention hashtag" rel="tag">#EnergyTransition</a>

Zane SelvansAs <a href="https://social.coop/tags/PUDL" class="mention hashtag" rel="tag">#PUDL</a> moves toward distributing only data (and much more of it) rather than expecting everyone to run the software (with its 500+ dependencies...) we're going to deprecate our output management layer.We see two possible deprecation paths. Should we go slow? Or rip the band-aid off now?<a href="https://social.coop/tags/OpenData" class="mention hashtag" rel="tag">#OpenData</a> <a href="https://social.coop/tags/EnergyTransition" class="mention hashtag" rel="tag">#EnergyTransition</a> <a href="https://social.coop/tags/OpenSource" class="mention hashtag" rel="tag">#OpenSource</a> <a href="https://social.coop/tags/EnergyMastodon" class="mention hashtag" rel="tag">#EnergyMastodon</a> <a href="https://social.coop/tags/datadon" class="mention hashtag" rel="tag">#datadon</a> <a href="https://social.coop/tags/pydata" class="mention hashtag" rel="tag">#pydata</a> <a href="https://social.coop/tags/EnergyTwitter" class="mention hashtag" rel="tag">#EnergyTwitter</a> Discussion on GitHub here: <a href="https://github.com/orgs/catalyst-cooperative/discussions/2503" target="_blank" rel="nofollow noopener noreferrer" translate="no">https://github.com/orgs/catalyst-cooperative/discussions/2503</a>

Zane SelvansI did not realize you can post up to 100GB of data to <a href="https://social.coop/tags/Kaggle" class="mention hashtag" rel="tag">#Kaggle</a> and they provide access to computational resources and <a href="https://social.coop/tags/Jupyter" class="mention hashtag" rel="tag">#Jupyter</a> notebooks.We're thinking about automatically posting all our <a href="https://social.coop/tags/PUDL" class="mention hashtag" rel="tag">#PUDL</a> data there, and maybe running community competitions to help solve entity matching, anomaly detection, and imputation problems. Is there any downside to doing this?<a href="https://social.coop/tags/OpenData" class="mention hashtag" rel="tag">#OpenData</a> <a href="https://social.coop/tags/MachineLearning" class="mention hashtag" rel="tag">#MachineLearning</a> <a href="https://social.coop/tags/DataScience" class="mention hashtag" rel="tag">#DataScience</a> <a href="https://social.coop/tags/EnergyTransition" class="mention hashtag" rel="tag">#EnergyTransition</a> <a href="https://social.coop/tags/EnergyTwitter" class="mention hashtag" rel="tag">#EnergyTwitter</a> <a href="https://social.coop/tags/EnergyMastodon" class="mention hashtag" rel="tag">#EnergyMastodon</a> <a href="https://www.kaggle.com/datasets/zaneselvans/catalyst-cooperative-pudl" target="_blank" rel="nofollow noopener noreferrer" translate="no">https://www.kaggle.com/datasets/zaneselvans/catalyst-cooperative-pudl</a>

Zane SelvansA few <a href="https://social.coop/tags/PUDL" class="mention hashtag" rel="tag">#PUDL</a> announcements!<a href="https://github.com/orgs/catalyst-cooperative/discussions/2475" target="_blank" rel="nofollow noopener noreferrer" translate="no">https://github.com/orgs/catalyst-cooperative/discussions/2475</a>Our migration to <a href="https://data-folks.masto.host/@dagster" class="u-url mention">@dagster</a> is progressing rapidly. If you use PUDL and run the ETL yourself, and need help getting Dagster set up, feel free to sign up for office hours:<a href="https://calendly.com/catalyst-cooperative/pudl-office-hours" target="_blank" rel="nofollow noopener noreferrer" translate="no">https://calendly.com/catalyst-cooperative/pudl-office-hours</a>Or ask for help in our GitHub discussions: <a href="https://github.com/orgs/catalyst-cooperative/discussions" target="_blank" rel="nofollow noopener noreferrer" translate="no">https://github.com/orgs/catalyst-cooperative/discussions</a><a href="https://social.coop/tags/OpenSource" class="mention hashtag" rel="tag">#OpenSource</a> <a href="https://social.coop/tags/OpenData" class="mention hashtag" rel="tag">#OpenData</a> <a href="https://social.coop/tags/EnergyTransition" class="mention hashtag" rel="tag">#EnergyTransition</a> <a href="https://social.coop/tags/EnergyMastodon" class="mention hashtag" rel="tag">#EnergyMastodon</a> <a href="https://social.coop/tags/datadon" class="mention hashtag" rel="tag">#datadon</a> <a href="https://social.coop/tags/DataEngineering" class="mention hashtag" rel="tag">#DataEngineering</a> <a href="https://social.coop/tags/EnergyTwitter" class="mention hashtag" rel="tag">#EnergyTwitter</a>

Zane SelvansWe finally have the whole <a href="https://social.coop/tags/PUDL" class="mention hashtag" rel="tag">#PUDL</a> data pipeline running in <a href="https://data-folks.masto.host/@dagster" class="u-url mention">@dagster</a> and the visualizations make it very clear where we need to parallelize stuff. 🐌 <a href="https://social.coop/tags/datadon" class="mention hashtag" rel="tag">#datadon</a> Anybody else running particularly large or complex <a href="https://social.coop/tags/OpenData" class="mention hashtag" rel="tag">#OpenData</a> / <a href="https://social.coop/tags/OpenSource" class="mention hashtag" rel="tag">#OpenSource</a> DAGs with these tools? We'd love to compare notes.It would also be cool if there were some way to expose all this information to our users in a read-only form, so they can see what's happening with the nightly builds too.

Zane SelvansThis kind of analysis has been done for the US ISO/RTO regions before, but this is the first publicly available analysis of non-ISO/RTO regions like the Southeast and West. The data illuminates massive opportunities for reduced reliance on coal and increased customer savings over time. A lot of the ongoing plant capital expenses and non-fuel O&M costs come from FERC Form 1 data liberated by <a href="https://social.coop/tags/CatalystCoop" class="mention hashtag" rel="tag">#CatalystCoop</a> <a href="https://social.coop/tags/PUDL" class="mention hashtag" rel="tag">#PUDL</a>

Zane SelvansCatalyst Cooperative has brought together a bunch of data for the Deployment Gap Education Fund to track anti-renewable energy policies at the county and local level, and understand how they intersect with and constrain deployment. Public Tableau dashboard here:<a href="https://public.tableau.com/app/profile/deployment.gap.model/viz/CleanEnergyandOpposition/ExistingvsProposed" target="_blank" rel="nofollow noopener noreferrer" translate="no">https://public.tableau.com/app/profile/deployment.gap.model/viz/CleanEnergyandOpposition/ExistingvsProposed</a><a href="https://social.coop/tags/EnergyTransition" class="mention hashtag" rel="tag">#EnergyTransition</a> <a href="https://social.coop/tags/EnergyMastodon" class="mention hashtag" rel="tag">#EnergyMastodon</a> <a href="https://social.coop/tags/EnergyTwitter" class="mention hashtag" rel="tag">#EnergyTwitter</a> <a href="https://social.coop/tags/ClimateChange" class="mention hashtag" rel="tag">#ClimateChange</a> <a href="https://social.coop/tags/Solar" class="mention hashtag" rel="tag">#Solar</a> <a href="https://social.coop/tags/CatalystCoop" class="mention hashtag" rel="tag">#CatalystCoop</a> <a href="https://social.coop/tags/PUDL" class="mention hashtag" rel="tag">#PUDL</a> <a href="https://social.coop/tags/OpenData" class="mention hashtag" rel="tag">#OpenData</a> <a href="https://social.coop/tags/datascience" class="mention hashtag" rel="tag">#datascience</a>

Zane SelvansThe Open Grid Emissions Initiative, which uses <a href="https://social.coop/tags/CatalystCoop" class="mention hashtag" rel="tag">#CatalystCoop</a> <a href="https://social.coop/tags/PUDL" class="mention hashtag" rel="tag">#PUDL</a> data as one of its main inputs, is trying to do this for historical analyses. Even if it's never useful for dispatching demand, it'll be hugely valuable for modeling the feasibility of 24/7 renewables.<a href="https://github.com/singularity-energy/open-grid-emissions" target="_blank" rel="nofollow noopener noreferrer" translate="no">https://github.com/singularity-energy/open-grid-emissions</a>

Zane SelvansYo, here is a new PUDL data release, including updates to all the data through the end of 2021. <a href="https://social.coop/tags/FERC" class="mention hashtag" rel="tag">#FERC</a> Form 1, <a href="https://social.coop/tags/EIA" class="mention hashtag" rel="tag">#EIA</a> 860/923, <a href="https://social.coop/tags/EPA" class="mention hashtag" rel="tag">#EPA</a> CEMS, etc. Tarball includes the <a href="https://social.coop/tags/SQLite" class="mention hashtag" rel="tag">#SQLite</a> DBs + <a href="https://social.coop/tags/ApacheParuqet" class="mention hashtag" rel="tag">#ApacheParuqet</a> for the CEMS, a <a href="https://social.coop/tags/Docker" class="mention hashtag" rel="tag">#Docker</a> container with the software environment used to create them, and some example <a href="https://social.coop/tags/Jupyter" class="mention hashtag" rel="tag">#Jupyter</a> notebooks!<a href="https://social.coop/tags/OpenData" class="mention hashtag" rel="tag">#OpenData</a> <a href="https://social.coop/tags/EnergyMastodon" class="mention hashtag" rel="tag">#EnergyMastodon</a> <a href="https://social.coop/tags/EnergyTwitter" class="mention hashtag" rel="tag">#EnergyTwitter</a> <a href="https://social.coop/tags/Energy" class="mention hashtag" rel="tag">#Energy</a> <a href="https://social.coop/tags/PyData" class="mention hashtag" rel="tag">#PyData</a> <a href="https://social.coop/tags/ClimateChange" class="mention hashtag" rel="tag">#ClimateChange</a> <a href="https://social.coop/tags/Policy" class="mention hashtag" rel="tag">#Policy</a> <a href="https://social.coop/tags/CatalystCoop" class="mention hashtag" rel="tag">#CatalystCoop</a> <a href="https://social.coop/tags/pudl" class="mention hashtag" rel="tag">#pudl</a> <a href="https://social.coop/tags/DataEngineering" class="mention hashtag" rel="tag">#DataEngineering</a> <a href="https://social.coop/tags/DataScience" class="mention hashtag" rel="tag">#DataScience</a> <a href="https://zenodo.org/record/7472137" target="_blank" rel="nofollow noopener noreferrer" translate="no">https://zenodo.org/record/7472137</a>

Zane SelvansWoo! Draft deployment of our extracted <a href="https://social.coop/tags/FERC" class="mention hashtag" rel="tag">#FERC</a> <a href="https://social.coop/tags/XBRL" class="mention hashtag" rel="tag">#XBRL</a> data using <a href="https://fedi.simonwillison.net/@simon" class="u-url mention">@simon</a>'s <a href="https://social.coop/tags/Datasette" class="mention hashtag" rel="tag">#Datasette</a>. Including 2021 data for FERC forms 1, 2, 6, 60, and 714. Still need to clean up the metadata, but hey it's a relational DB usable for analysis and transformation... not a bunch-o-gigabytes of XML. <a href="https://social.coop/tags/sqlite" class="mention hashtag" rel="tag">#sqlite</a> <a href="https://social.coop/tags/OpenData" class="mention hashtag" rel="tag">#OpenData</a> <a href="https://social.coop/tags/EnergyTwitter" class="mention hashtag" rel="tag">#EnergyTwitter</a> <a href="https://social.coop/tags/EnergyMastodon" class="mention hashtag" rel="tag">#EnergyMastodon</a> <a href="https://social.coop/tags/PUDL" class="mention hashtag" rel="tag">#PUDL</a> <a href="https://data.catalyst.coop/" target="_blank" rel="nofollow noopener noreferrer" translate="no">https://data.catalyst.coop/</a>

Zane SelvansThe 2021 <a href="https://social.coop/tags/FERC" class="mention hashtag" rel="tag">#FERC</a> Form 1 data has been more recalcitrant, since they've switched to using <a href="https://social.coop/tags/XBRL" class="mention hashtag" rel="tag">#XBRL</a> for reporting (after 27 years of Visual FoxPro...), but we're close!It looks like there's enough structured information in the XBRL taxonomies that we can reproduce all the calculations and tag the data with the relevant FERC accounting categories.<a href="https://social.coop/tags/ferc1" class="mention hashtag" rel="tag">#ferc1</a> <a href="https://social.coop/tags/utility" class="mention hashtag" rel="tag">#utility</a> <a href="https://social.coop/tags/finance" class="mention hashtag" rel="tag">#finance</a> <a href="https://social.coop/tags/pudl" class="mention hashtag" rel="tag">#pudl</a>

Recent searches

Search options

Administered by:

Server stats:

#pudl