Added files

README.md

@@ -1,14 +1,13 @@
 ---
-title: Poc
-emoji: 💻
-colorFrom: gray
-colorTo: gray
+title: Trend Prediction App
+emoji: 🚀
+colorFrom: indigo
+colorTo: pink
 sdk: streamlit
-sdk_version: 1.39.0
+sdk_version: 1.37.1
 app_file: app.py
 pinned: false
 license: apache-2.0
-short_description: Bioma AI Prototype
 ---
 
 Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference

app.py

@@ -0,0 +1,6 @@
+import streamlit as st
+
+pg = st.navigation({"Bioma AI PoC":[st.Page("page_1.py", title="Semantic Analysis"), 
+                    st.Page("page_2.py", title="Price Forecasting"),
+                    st.Page("page_3.py", title="MLOps Pipeline")]})
+pg.run()

convert_first.csv

@@ -0,0 +1,150 @@
+,headline,topic_verification
+0,SPAIN DATA: H1 crude imports rise 11% to 1.4 million b/d,Crude Oil
+1,REFINERY NEWS: Host of Chinese units back from works; Jinling maintenance in Nov-Dec,Macroeconomic & Geopolitics
+2,REFINERY NEWS ROUNDUP: Mixed runs in Asia-Pacific,Macroeconomic & Geopolitics
+3,Physical 1%S fuel oil Med-North spread hits record high on competitive bidding in Platts MOC,Middle Distillates
+4,"Indian ports see Jan-July bunker, STS calls up 64% on year, monsoon hits July demand",Macroeconomic & Geopolitics
+5,LNG bunker prices in Europe hit 8-month high amid rising demand,Light Ends
+6,REFINERY NEWS: Wellbred Trading acquires La Nivernaise de Raffinage in France,Heavy Distillates
+7,Wellbred Trading buys French diesel refinery that runs on used cooking oil,Macroeconomic & Geopolitics
+8,EU's climate monitor says 2024 'increasingly likely' to be warmest year on record,Macroeconomic & Geopolitics
+9,"European LPG discount to naphtha narrows, shifting petchem feedstock appetite",Light Ends
+10,REFINERY NEWS: Thai Oil’s Q2 utilization drops on planned CDU shutdown,Crude Oil
+11,South Korea’s top oil refiner SK Innovation joins carbon storage project in Australia,Middle Distillates
+12,CRUDE MOC: Middle East sour crude cash differentials hit month-to-date highs,Crude Oil
+13,CNOOC approves 100 Bcm of proven reserves at South China Sea gas field,Crude Oil
+14,"Singapore to work with Shell’s refinery, petrochemicals asset buyers to decarbonize: minister",Light Ends
+15,BLM federal Montana-Dakotas oil and gas lease sale nets nearly $24 mil: Energynet.com,Middle Distillates
+16,REFINERY NEWS: Oman's Sohar undergoes unplanned shutdown: sources,Crude Oil
+17,"OIL FUTURES: Crude prices higher as US stockpiles extend decline, demand concerns cap gains",Crude Oil
+18,Qatar announces acceptance of Sep LPG cargoes with no cuts or delays heard,Light Ends
+19,"South Korea aims for full GCC FTA execution by year-end, refiners hopeful for cheaper sour crude",Crude Oil
+20,"Indonesia sets Minas crude price at $84.95/b for July, rising $3.35/b from June",Crude Oil
+21,Cathay Pacific H1 2024 passenger traffic rises 36% on year; Hong Kong’s jet fuel demand bolstered,Middle Distillates
+22,US DATA: Total ULSD stocks near a six-month high as demand continues to fall,Middle Distillates
+23,US DATA: Product supplied of propane and propylene reach three-month high,Light Ends
+24,Internatonal Seaways focused on replacing aging fleet during second quarter: CEO,Crude Oil
+25,"Devon Energy's oil output hits all-time record high from Delaware, Eagle Ford operations",Crude Oil
+26,"Brazil's Prio still waiting on IBAMA license approvals to boost oil, gas output",Crude Oil
+27,REFINERY NEWS: Delek US sees Q3 refinery utilization dip from record Q2 highs,Light Ends
+28,"OIL FUTURES: Crude rallies as traders eye tighter US supply, global financial market stabilization",Crude Oil
+29,"Prompt DFL, CFD contracts rally",Crude Oil
+30,"REFINERY NEWS: Petroperú sees 2Q refined fuel sales drop 4.4% on year to 93,700 b/d",Middle Distillates
+31,W&T Offshore nears close of new US Gulf of Mexico drilling joint venture,Crude Oil
+32,"Imrproved efficiencies, continued M&A activity to drive growth for Permian Resources",Heavy Distillates
+33,Mexico's Pemex to explore deposit adjacent to major onshore gas field Quesqui,Middle Distillates
+34,REFINERY NEWS: Par Pacific reports softer south Rockies results as Midwest barrels spill into region,Middle Distillates
+35,"Suncor sees improved H2 oil and gas output, completes major Q2 turnarounds",Middle Distillates
+36,"Brazil's Petrobras, Espirito Santo state to study potential CCUS, hydrogen hubs",Middle Distillates
+37,"Argentina raises biodiesel, ethanol prices for blending by 1.5% in August",Middle Distillates
+38,Bolivia offers tax breaks to import equipment for biodiesel plants following fuel shortages,Light Ends
+39,"US DATA: West Coast fuel oil stocks hit a six-week low, EIA says",Middle Distillates
+40,Iraq’s SOMO cuts official selling prices for September-loading crude oil for Europe,Crude Oil
+41,Nigeria's Dangote refinery plans to divest 12.75% stake: ratings agency,Middle Distillates
+42,REFINERY NEWS: Kazakhstan's Atyrau processes 2.9 mil mt crude in H1,Middle Distillates
+43,REFINERY NEWS: Thailand's IRPC reports Q2 utilization of 94%,Light Ends
+44,DNO reports higher Q2 crude production in Iraq's Kurdish region,Crude Oil
+45,"ADNOC L&S expects ‘strong rates’ in tankers, dry-bulk, containers in 2024",Crude Oil
+46,WAF crude tanker rates hit 10-month lows amid sluggish inquiry levels,Crude Oil
+47,Senegal's inaugural crude stream Sangomar to load 3.8 mil barrels in September,Crude Oil
+48,China's July vegetable oil imports rise 3% on month as buyers replenish domestic stocks,Macroeconomic & Geopolitics
+49,CRUDE MOC: Middle East sour crude cash differentials rebound,Crude Oil
+50,OIL FUTURES: Crude oil recovers as financial markets improve,Crude Oil
+51,"Tullow sees rise in crude output, profits on-year in H1 2024",Crude Oil
+52,Russia's Taman port June-July oil products throughput up 26% on year,Heavy Distillates
+53,JAPAN DATA: Oil product exports rise 4.5% on week to 2.42 mil barrels,Crude Oil
+54,REFINERY NEWS: Petro Rabigh to be upgraded after Aramco takes control,Crude Oil
+55,Canada's ShaMaran closes acquisition of Atrush oil field,Crude Oil
+56,CHINA DATA: July natural gas imports rise 5% on year to 10.9 mil mt,Light Ends
+57,"OIL FUTURES: Crude stabilizes on technical bounce, supply uncertainty",Crude Oil
+58,JAPAN DATA: Oil product stocks rise 0.8% on week to 55.32 mil barrels,Crude Oil
+59,Japan cuts Aug 8-14 fuel subsidy by 21% as crude prices drop,Middle Distillates
+60,JAPAN DATA: Refinery runs rise to 67% over July 28-Aug 3 on higher crude throughput,Light Ends
+61,Asian reforming spread hits over two-year low as gasoline prices lag naphtha,Light Ends
+62,Asia medium sulfur gasoil differential weakens as Indonesia demand tapers,Middle Distillates
+63,"QatarEnergy raises Sep Land, Marine crude OSPs by 45-75 cents/b from Aug",Heavy Distillates
+64,ADNOC sets Murban Sep OSP $1.28/b higher on month at $83.80/b,Heavy Distillates
+65,"Diamondback Energy keeps pushing well drilling, completion efficiencies in Q2",Middle Distillates
+66,"Genel Energy’s oil production from Tawke field increases to 19,510 b/d in 1H 2024",Middle Distillates
+67,Longer laterals and higher well performance drive Rocky Mountain production: Oneok,Light Ends
+68,US DOE seeks to buy 3.5 million barrels of crude for delivery to SPR in January 2025,Crude Oil
+69,"FPSO Maria Quiteria arrives offshore Brazil, to reduce emissions: Petrobras",Middle Distillates
+70,OIL FUTURES: Crude edges higher as market stabilizes amid Middle Eastern supply concerns,Crude Oil
+71,"US EIA lowers 2024 oil price outlook by $2/b, but still predicts increases",Crude Oil
+72,"Shell, BP to fund South Africa's Sapref refinery operations in government takeover",Light Ends
+73,"Indian Oil cancels tender to build a 10,000 mt/yr renewable hydrogen plant",Light Ends
+74,"Brazil's Prio July oil equivalent output falls 31.7% on maintenance, shuttered wells",Crude Oil
+75,Eni follows Ivory Coast discoveries with four new licenses,Crude Oil
+76,EU DATA: MY 2024-25 soybean meal imports rise 8% on year as of Aug 4,Macroeconomic & Geopolitics
+77,"Greek PPC to buy a 600 MW Romanian wind farm, portfolio from Macquarie-owned developer",Macroeconomic & Geopolitics
+78,Vitol to take Italian refiner Saras private after acquiring 51% stake,Macroeconomic & Geopolitics
+79,Mediterranean sweet crude market shows muted response to Sharara shutdown,Macroeconomic & Geopolitics
+80,REFINERY NEWS: Vitol acquires 51% in Italian refiner Saras,Macroeconomic & Geopolitics
+81,Rotterdam LNG bunkers spread with VLSFO narrows to 2024 low,Light Ends
+82,Argentina’s YPF finds buyers for 15 maturing conventional blocks as it focuses on Vaca Muerta,Heavy Distillates
+83,REFINERY NEWS ROUNDUP: Nigerian plants in focus,Macroeconomic & Geopolitics
+84,"REFINERY NEWS: Valero shuts CDU, FCCU at McKee refinery for planned work",Macroeconomic & Geopolitics
+85,Kazakhstan extends ban on oil products exports by truck for six months,Macroeconomic & Geopolitics
+86,Physical Hi-Lo spread hits 3 month high amid prompt LSFO demand,Heavy Distillates
+87,CRUDE MOC: Middle East sour crude cash differentials slip to fresh lows,Crude Oil
+88,"Nigeria launches new Utapate crude grade, first cargo heads to Spain",Crude Oil
+89,REFINERY NEWS: Turkish Tupras Q2 output rises 15% on the quarter and year,Middle Distillates
+90,"CHINA DATA: Independent refineries’ Iranian crude imports fall in July, ESPO inflows rebound",Crude Oil
+91,Gunvor acquires TotalEnergies' 50% stake in Pakistan retail fuel business,Middle Distillates
+92,INTERVIEW: Coal to remain a dominant power source in India: Menar MD,Macroeconomic & Geopolitics
+93,OIL FUTURES: Crude price holds steady as demand expectations cap gains,Crude Oil
+94,Fujairah’s HSFO August HSFO ex-wharf premiums slip; stocks adequate,Heavy Distillates
+95,JAPAN DATA: US crude imports more than double in March as Middle East dependency eases,Crude Oil
+96,Dubai crude futures traded volume on TOCOM rebounds in July from record low,Crude Oil
+97,Japan's spot electricity price retreats 8% as temperatures ease,Macroeconomic & Geopolitics
+98,"HONG KONG DATA: June oil product imports surge 32% on month to 226,475 barrels",Crude Oil
+99,NextDecade signs contract with Bechtel to build Rio Grande LNG expansion,Light Ends
+100,"Kosmos sees 2024 total output of 90,000 boe/d, despite Q2 operations thorns: CEO",Crude Oil
+101,"Dated Brent reaches two-month low Aug. 5 as physical, derivatives prices slide on day",Middle Distillates
+102,"Alaska North Slope crude output up in July, but long-term decline continues",Crude Oil
+103,Balance-month DFL contract slips to seven-week low in bearish sign for physical crude fundamentals,Crude Oil
+104,Iraqi Kurdistan officials order crackdown on illegal refineries over pollution,Macroeconomic & Geopolitics
+105,Rhine barge cargo navigation limits set to kick in amid dryer weather,Middle Distillates
+106,Bolivia returns diesel supplies to normal following shortages,Middle Distillates
+107,OCI optimistic about methanol demand driven by decarbonization efforts,Light Ends
+108,Mitsubishi to supply turbine for 30% hydrogen co-firing in Malaysia power plant,Middle Distillates
+109,ATLANTIC LNG: Key market indicators for Aug. 5-9,Light Ends
+110,"Eurobob swap, gas-nap spread falls below 6-month low amid crude selloff",Light Ends
+111,EMEA PETROCHEMICALS: Key market indicators for Aug 5-9,Light Ends
+112,EMEA LIGHT ENDS: Key market indicators for Aug 5 – 9,Light Ends
+113,EUROPE AND AFRICA RESIDUAL AND MARINE FUEL: Key market indicators Aug 5-9,Heavy Distillates
+114,TURKEY DATA: June crude flows via BTC pipeline up 8.1% on month,Crude Oil
+115,EMEA AGRICULTURE: Key market indicators for Aug 5–9,Macroeconomic & Geopolitics
+116,OIL FUTURES: Crude oil faces downward pressure amid wider weakness in financial markets,Crude Oil
+117,Woodside to acquire OCI’s low carbon ammonia project with CO2 capture in US,Middle Distillates
+118,Maire secures feasibility study for sustainable aviation fuel project in Indonesia,Middle Distillates
+119,CRUDE MOC: Middle East sour crude cash differentials plunge on risk-off sentiment,Middle Distillates
+120,"Zhoushan LSFO storage availability rises for 3rd month in Aug, hits record high",Middle Distillates
+121,Oil storage in Russia's Rostov region hit by drone strike,Macroeconomic & Geopolitics
+122,WAF TRACKING: Nigerian crude exports to Netherlands top 5-year high in July,Crude Oil
+123,"Vietnam’s Hai Linh receives license to import, export LNG",Light Ends
+124,Japan's Idemitsu could restart Tokuyama steam cracker on Aug 11,Light Ends
+125,Indonesia's biodiesel output up 12% in H1 on increased domestic mandates: APROBI,Middle Distillates
+126,CHINA DATA: Independent refiners' July feedstocks imports hit 3-month low at 3.65 mil b/d,Light Ends
+127,"Singapore’s Aug ex-wharf term LSFO premiums rise, demand moderate",Heavy Distillates
+128,"OIL FUTURES: Crude slumps as market volatility rages on recession, Middle East risks",Crude Oil
+129,Pakistan's HSFO exports nearly triple as focus shifts to cheaper power sources,Heavy Distillates
+130,"TAIWAN DATA: June oil products demand falls 3% on month to 758,139 b/d",Light Ends
+131,REFINERY NEWS: Japan's Cosmo restarts No. 1 Chiba CDU after glitches,Crude Oil
+132,ASIA PETROCHEMICALS: Key market indicators for Aug 5-9,Light Ends
+133,DME Oman crude futures traded volume rises for seventh straight month in July,Crude Oil
+134,ICE front-month Singapore gasoline swaps open interest rises 14.6% on month in July,Light Ends
+135,ASIA OCTANE: Key market indicators for Aug 5-9,Light Ends
+136,ICE Dubai crude futures July total traded volume rises 11.4% on month,Crude Oil
+137,"Lower-than-expected Aramco Sep OSPs a nod to weak Asian market, OPEC+ cut unwind",Crude Oil
+138,ASIA CRUDE OIL: Key market indicators for Aug 5-8,Crude Oil
+139,ASIA LIGHT ENDS: Key market indicators for Aug 5-8,Light Ends
+140,China fuel oil quotas decline seen supporting Q3 LSFO premiums in Zhoushan,Middle Distillates
+141,South Korea's short-term diesel demand under pressure on e-commerce firms' bankruptcy,Middle Distillates
+142,ICE front-month Singapore 10 ppm gasoil swap open interest rebounds 2% on month in July,Middle Distillates
+143,Saudi Aramco maintains or raises Asia-bound Sep crude OSPs by 10-20 cents/b,Crude Oil
+144,ASIA MIDDLE DISTILLATES: Key market indicators for Aug 5-8,Middle Distillates
+145,ICE front-month Singapore HSFO open interest rises 19.6% on month in July,Heavy Distillates
+146,REFINERY NEWS: Fort Energy at Fujairah ‘remains operational’,Macroeconomic & Geopolitics
+147,Container ship Groton attacked near Yemen amid growing Middle East security risks,Macroeconomic & Geopolitics
+148,Oil depot in Russia’s Belgorod region hit by drone strike,Macroeconomic & Geopolitics

data/topresults_platts_09082024_clean.csv

@@ -0,0 +1,12 @@
+,body,headline,updatedDate,topic_prediction,topic_verification,negative_score,neutral_score,positive_score,trend_prediction,trend_verification
+0," OPEC+ crude production in July made its biggest jump in almost a year, as Iraq and Kazakhstan raised their output despite committing to deeper cuts, while Russia also remained well over its quota.   The group's overall production was up 160,000 b/d compared with June, totaling 41.03 million b/d, the Platts OPEC+ survey from S&P Global Commodity Insights showed Aug. 8.   Member countries with quotas produced 437,000 b/d above target in July, up from 229,000 b/d in June.   July was the first month of compensation plans introduced by three countries that overproduced in the first half of 2024. Iraq pledged to cut an additional 70,000 b/d in July and Kazakhstan pledged to cut a further 18,000 b/d. Russia's compensation plan does not include additional cuts until October 2024.   The survey found that Iraq produced 4.33 million b/d in July, 400,000 b/d above its quota. This contributed to growth in OPEC production of 130,000 b/d to 26.89 million b/d.   Non-OPEC producers added a further 14.14 million b/d, up 30,000 b/d month on month.   This was driven by Kazakhstan, which increased output by 30,000 b/d. It is now producing 120,000 b/d above quota, taking into account its compensation cut.   Russia is also producing above quota, with output at 9.10 million b/d in July, against a quota of 8.98 million b/d.       The overproducers are part of a group that is implementing a combined 2.2 million b/d of voluntary cuts, currently in place until the end of the third quarter. The group then plans to gradually bring some of those barrels back to market from September if conditions allow. A further 3.6 million b/d of group-wide cuts are in place until the end of 2025.   The rise in output in July came despite the poor performance of the alliance's African contingent, with production in Nigeria, South Sudan, Gabon and Libya falling by a collective 80,000 b/d.   Pressure on overproducers has increased in recent weeks, as recession fears have driven oil prices below $80/b. Platts, part of Commodity Insights, assessed Dated Brent at $79.91/b Aug. 7.   A long-awaited rise in Chinese demand and high production from non-OPEC countries in the Americas -- including the US, Canada, Brazil and Guyana -- have also weakened prices in recent months.   OPEC+ has pledged to stick to its strategy of major production cuts through the third quarter, before gradually bringing barrels back to market. Overproduction and depressed oil prices threaten these plans.   The next meeting of the Joint Ministerial Monitoring Committee overseeing the agreement, which is co-chaired by Saudi Arabia and Russia, is scheduled for Oct. 2. A full ministerial meeting is scheduled for Dec. 1.   The Platts survey measures wellhead production and is compiled using information from oil industry officials, traders and analysts, as well as by reviewing proprietary shipping, satellite and inventory data.          OPEC+ crude production (million b/d)      OPEC-9  July-24  Change  June-24  Quota  Over/under    Algeria  0.90  0.00  0.90  0.908  -0.008    Congo-Brazzaville  0.26  0.00  0.26  0.277  -0.017    Equatorial Guinea  0.05  0.00  0.05  0.070  -0.020    Gabon  0.21  -0.01  0.22  0.169  0.041    Iraq*†  4.33  0.11  4.22  3.930  0.400    Kuwait  2.42  0.00  2.42  2.413  0.007    Nigeria  1.46  -0.04  1.50  1.500  -0.040    Saudi Arabia  8.99  0.01  8.98  8.978  0.012    UAE  2.99  0.02  2.97  2.912  0.078    TOTAL OPEC-9  21.61  0.09  21.52  21.157  0.453            OPEC EXEMPT   Change   Quota  Over/under    Iran  3.20  0.00  3.20  N/A  N/A    Libya  1.15  -0.01  1.16  N/A  N/A    Venezuela  0.93  0.05  0.88  N/A  N/A    TOTAL OPEC-12  26.89  0.13  26.76  N/A  N/A            NON-OPEC WITH QUOTAS   Change   Quota  Over/under    Azerbaijan  0.49  0.01  0.48  0.551  -0.061    Bahrain  0.18  0.00  0.18  0.196  -0.016    Brunei  0.07  0.01  0.06  0.083  -0.013    Kazakhstan†  1.57  0.03  1.54  1.450  0.120    Malaysia  0.35  0.00  0.35  0.401  -0.051    Oman  0.76  0.00  0.76  0.759  0.001    Russia  9.10  0.00  9.10  8.978  0.122    Sudan  0.03  0.00  0.03  0.064  -0.034    South Sudan  0.04  -0.02  0.06  0.124  -0.084    TOTAL NON-OPEC WITH QUOTAS  12.59  0.03  12.56  12.606  -0.016            NON-OPEC EXEMPT   Change   Quota  Over/under    Mexico  1.55  0  1.55  N/A  N/A    TOTAL NON-OPEC  14.14  0.03  14.11  N/A  N/A            OPEC+ MEMBERS WITH QUOTAS   Change   Quota  Over/under    TOTAL  34.20  0.12  34.08  33.76  0.437            OPEC+   Change   Quota  Over/under    TOTAL  41.03  0.16  40.87  N/A  N/A            * Includes estimated 250,000 b/d production in the semi-autonomous Kurdistan region of Iraq    † Iraq and Kazakhstan quotas reduced in line with compensation plans    Source: Platts OPEC+ survey by S&P Global Commodity Insights          ","OPEC+ produces 437,000 b/d above quota in first month of compensation cuts",2024-08-08 17:36:29+00:00,Macroeconomic & Geopolitics,Macroeconomic & Geopolitics,0.9936981650538123,0.03949102865047352,0.07103689164109918,Bearish,Bearish
+1,nan,"Non-OPEC July output up 30,000 b/d at 14.14 mil b/d: Platts survey",2024-08-08 14:00:12+00:00,Macroeconomic & Geopolitics,Macroeconomic & Geopolitics,0.9932350855162315,0.024123551368425825,0.12366691833078211,Bearish,Bearish
+2,nan,"OPEC+ producers with quotas 437,000 b/d above target in July: Platts survey",2024-08-08 14:00:11+00:00,Macroeconomic & Geopolitics,Macroeconomic & Geopolitics,0.9936222216140704,0.048969414364614584,0.06152339592702592,Bearish,Bearish
+3,nan,"OPEC crude output up 130,000 b/d at 26.89 mil b/d in July: Platts survey",2024-08-08 14:00:11+00:00,Macroeconomic & Geopolitics,Macroeconomic & Geopolitics,0.9933905710185933,0.03299545514176609,0.10238077772498969,Bearish,Bearish
+4,nan,"OPEC+ July crude output up 160,000 b/d at 41.03 mil b/d: Platts survey",2024-08-08 14:00:10+00:00,Macroeconomic & Geopolitics,Macroeconomic & Geopolitics,0.9870346671527929,0.02437343317152671,0.2153639976093806,Bearish,Bearish
+5,nan,"Iraq, Russia, Kazakhstan overproduce in first month of compensation cuts: Platts survey",2024-08-08 14:00:10+00:00,Crude Oil,Crude Oil,0.6294152478714086,0.06749551758023337,0.927845667131505,Bullish,Bullish
+6," UK-based upstream producer Harbour Energy plans to start its new Talbot oil and gas tie-in project at the J-Area hub in the North Sea by the end of 2024, boosting Ekofisk blend volumes, it said Aug. 8.   Harbour, in a statement, reported a 19% year-on-year drop in its UK oil and gas production in the first half of 2024 to 149,000 b/d of oil equivalent. It noted a significant maintenance impact, including a planned shutdown in June at the J-Area, which sends oil and gas to Teesside, with the liquids loaded as Ekofisk blend. Ekofisk is a component in the Platts Dated Brent price assessment process.   Talbot, a multiwell development, is expected to recover 18 million boe of light oil and gas over 16 years. It will add to oil volumes flowing through the J-Area into the Norpipe route to Teesside, contributing to the predominantly Norwegian Ekofisk blend.   Harbour also flagged an ongoing maintenance impact on production through much of the Q3 2024, including a 40-day shutdown at the Britannia hub starting in August, which will impact flows into the Forties blend. The maintenance was expected to start in the next few days and be completed in September, according to a source close to the situation. Britannia was also expected to be impacted by a four-week shutdown of the SAGE gas pipeline  starting Aug. 27 .   Harbour has made ""good progress to date on the maintenance shutdowns and our UK capital projects, which are on track to materially increase production in the fourth quarter,"" it said.   The North Sea typically sees a drop in production volumes in the summer due to maintenance.    Non-UK diversification    Harbour reiterated its efforts to diversify away from the UK, with an acquisition of Wintershall Dea assets underway, having strongly objected to punitive tax rates. It said its overall effective tax rate in the first half of 2024 was 85%, partly reflecting not-fully deductible costs under the UK tax regime.   Harbour reported 10,000 boe/d of additional production outside the UK in the first half of the year, in Indonesia and Vietnam. It noted progress in Mexico, where Front End Engineering and Design has begun for the Zama oil project, estimated at 700 million barrels of light crude. Harbour is set to increase its Zama stake from 12% to 32% following the Wintershall acquisition.   In the first half of 2024 ""we made significant progress towards completing the Wintershall Dea acquisition, which is now expected early in the fourth quarter,"" CEO Linda Cook said.   ""The acquisition will transform the scale, geographical diversity and longevity of our portfolio and strengthen our capital structure, enabling us to deliver enhanced shareholder returns over the long run while also positioning us for further opportunities.”   Platts Dated Brent was assessed at $79.91/b on Aug. 8, up $3.64 on the day. Platts is part of S&P Global Commodity Insights. ",UK's Harbour Energy says on track with North Sea Talbot oil tie-in,2024-08-08 13:54:45+00:00,Crude Oil,Crude Oil,0.31882130542268583,0.04218598724364094,0.9882147680155492,Bullish,Bullish
+7," The INPEX-operated Ichthys LNG project in Australia has recovered to an 85% overall production rate after Train 2 restarted on July 28 following an outage on July 20 that was caused by a glitch, an INPEX spokesperson told S&P Global Commodity Insights Aug. 8.   Currently, the onshore Ichthys LNG plant is running at 100% at Train 1, and about 70% at Train 2, putting the overall production rate at about 85%, the spokesperson said.   The Ichthys LNG project is slated to resume full runs in October, when it plans to carry out some scheduled maintenance work lasting around a week, the spokesperson said.   INPEX has estimated that fewer than five LNG cargoes of Ichthys LNG shipments will be affected as a result of the glitch, the spokesperson said.   However, the INPEX spokesperson declined to elaborate on actual production volumes at the Ichthys LNG plant, which has yet to reach its operational capacity of 9.3 million mt/year.   INPEX has been building a framework for a stable supply of 9.3 million mt/year of LNG at its operated Ichthys project by debottlenecking the facility, upgrading the cooling systems for liquefication and taking measures to address vibration issues.   As of July, the Ichthys project has shipped a total of 76 LNG cargoes this year, with July shipments having slipped to 10 cargoes from 11 cargoes in June.   Ichthys LNG shipments will slow to 10 cargoes per month in the second half of 2024, the spokesperson said, compared with an average of 11 cargoes per month in the first half of the year.   In the first seven months of the year the Ichthys project shipped 14 plant condensate cargoes, 18 field condensate cargoes and 20 LPG cargoes.   In the January-June period INPEX produced 662,000 b/d of oil equivalent, and it now expects its 2024 production to be 644,800 boe/d, down from its May outlook of 645,300 boe/d for the year as a result of the Ichthys LNG production issues, the spokesperson said.   The project, operated by INPEX with 67.82%, involves piping gas from the offshore Ichthys field in the Browse Basin in Northwestern Australia more than 890 km (552 miles) to the onshore LNG plant near Darwin, which has an 8.9 million mt/year nameplate capacity. At peak, it has the capacity to produce 1.65 million mt/year of LPG and 100,000 b/d of condensate. ",Australia's Ichthys LNG recovers 85% output after Train 2 outage; to recover full runs in Oct,2024-08-08 11:53:44+00:00,Other,Other,0.770051212604236,0.010564989240227092,0.9773946433377442,Bullish,Bullish
+8," NTPC Limited, India’s largest power generation utility, has partnered with LanzaTech to implement carbon recycling technology at its new facility in central India, in a significant move towards sustainable energy.   The project will convert CO2 emissions and green hydrogen into ethanol using LanzaTech's second-generation bioreactor, the US-based company said in a statement Aug. 7.   NTPC's upcoming plant will be the first in India to deploy this advanced technology, which captures carbon-rich gases before they enter the atmosphere.   The LanzaTech bioreactor uses proprietary microbes to transform these gases into sustainable fuels, chemicals, and raw materials. The microbes convert CO2 and H2 into ethanol, a critical component for producing green energy products such as sustainable aviation fuels (SAF) and renewable diesel.   This in turn boosts NTPC's goals by producing ethanol from waste-based feedstocks, promoting a circular carbon economy.   According to the statement, the project was conceptualized and designed in collaboration with NTPC's research and development arm, NETRA (NTPC Energy Technology Research Alliance).   The facility aims to demonstrate the commercial viability of LanzaTech’s technology in producing ethanol from waste-based feedstocks by leveraging CO2 as sole carbon source.   Jakson Green, a new energy firm, is responsible for development of this Chhattisgarh-based facility, handling from design and engineering to procurement and construction.   This first-of-its-kind plant is projected to abate 7,300 mt/year of CO2 annually, equivalent to the carbon sequestered by 8,523 acres of forest land.   The carbon and hydrogen to renewable ethanol facility is slated to begin operations within two years.   Dr. Jennifer Holmgren, CEO of LanzaTech, emphasized the strategic importance of this partnership, stating, “Our collaboration with NTPC and Jakson Green sets a roadmap for the commercial deployment of CO2 as a key feedstock.”   Jakson Green is already developing India’s largest green hydrogen fueling station and a low-carbon methanol plant for leading government companies.   LanzaTech technology is also being used at various other operations in India, producing ethanol at Indian Oil Corporation’s Panipat facility which will also be used for SAF. The company has also partnered with GAIL and Mangalore Refinery and Petrochemicals Limited on similar projects.   Platts, part of Commodity Insights, assessed SAF production costs (palm fatty acid distillate) in Southeast Asia at $1,589.91/mt Aug. 7, down $19.50/mt from the previous assessment. ",NTPC advances clean energy goals with LanzaTech CO2-to-ethanol technology,2024-08-08 11:30:49+00:00,Light Ends,Light Ends,0.21314498348994937,0.11135607578700647,0.9908829648109232,Bullish,Bullish
+9," UAE-based Dana Gas said it expects to resume drilling activities in Egypt after the country’s parliament ratified a law to consolidate its concessions to operate in the country under a new concession with Egyptian Natural Gas Holding Co.   The new agreement ratified by the Egyptian parliament was already approved by the Egyptian Cabinet in March, authorizing the country’s minister of oil and Egyptian Natural Gas to finalize a new concession agreement with Dana Gas, the company said in an Aug. 8 statement. Since 2001, Dana Gas has been in discussions with Egyptian Natural Gas to consolidate three of its four concessions into a new concession with improved terms, according to Dana Gas’s website.   “The revised terms should enable meaningful future investments alongside a resumption of drilling activities, positively impacting the company’s production levels in Egypt and helping the country meet its growing gas demand,” Dana Gas said in the statement.   Egypt has halted LNG exports during the summer months and has turned to LNG imports instead to meet high seasonal demand amid declining domestic production. The development comes as delivered spot LNG prices to the East Mediterranean continue to trade above $10/MMBtu.   Platts, part of S&P Global Commodity Insights, assessed the DES LNG East Mediterranean marker at $12.47/MMBtu Aug. 7, the highest since the assessment started in December 2023.   The company’s first-half 2024 production in Egypt was 59,800 boe/d, down 25% from the same period a year earlier, mostly due to natural field declines, according to the statement. Dana Gas did not state when it expects to bring new production streams online in the country.   Dana Gas's production in the Kurdish region of northern Iraq increased 3% over the same period to 37,600 boe/d due to increased demand for gas from local power plants, the company said. ",Dana Gas expects to resume drilling activities in Egypt after new concession,2024-08-08 11:23:48+00:00,Other,Other,0.023988005652641385,0.7891432360374782,0.9608502193290972,Bullish,Bullish
+10,nan,"Indonesia sets Minas crude price at $84.95/b for July, rising $3.35/b from June",2024-08-08 01:41:13+00:00,Middle Distillates,Middle Distillates,0.9926734319450401,0.04286090006550804,0.07892061673161296,Bearish,Bearish

experimentation_mlops/example/MLProject

@@ -0,0 +1,36 @@
+name: multistep_example
+
+python_env: python_env.yaml
+
+entry_points:
+  load_raw_data:
+    command: "python load_raw_data.py"
+
+  etl_data:
+    parameters:
+      ratings_csv: path
+      max_row_limit: {type: int, default: 100000}
+    command: "python etl_data.py --ratings-csv {ratings_csv} --max-row-limit {max_row_limit}"
+
+  als:
+    parameters:
+      ratings_data: path
+      max_iter: {type: int, default: 10}
+      reg_param: {type: float, default: 0.1}
+      rank: {type: int, default: 12}
+    command: "python als.py --ratings-data {ratings_data} --max-iter {max_iter} --reg-param {reg_param} --rank {rank}"
+
+  train_keras:
+    parameters:
+      ratings_data: path
+      als_model_uri: string
+      hidden_units: {type: int, default: 20}
+    command: "python train_keras.py --ratings-data {ratings_data} --als-model-uri {als_model_uri} --hidden-units {hidden_units}"
+
+  main:
+    parameters:
+      als_max_iter: {type: int, default: 10}
+      keras_hidden_units: {type: int, default: 20}
+      max_row_limit: {type: int, default: 100000}
+    command: "python main.py --als-max-iter {als_max_iter} --keras-hidden-units {keras_hidden_units}
+                             --max-row-limit {max_row_limit}"

experimentation_mlops/example/als.py

@@ -0,0 +1,69 @@
+"""
+Trains an Alternating Least Squares (ALS) model for user/movie ratings.
+The input is a Parquet ratings dataset (see etl_data.py), and we output
+an mlflow artifact called 'als-model'.
+"""
+import click
+import pyspark
+from pyspark.ml import Pipeline
+from pyspark.ml.evaluation import RegressionEvaluator
+from pyspark.ml.recommendation import ALS
+
+import mlflow
+import mlflow.spark
+
+
+@click.command()
+@click.option("--ratings-data")
+@click.option("--split-prop", default=0.8, type=float)
+@click.option("--max-iter", default=10, type=int)
+@click.option("--reg-param", default=0.1, type=float)
+@click.option("--rank", default=12, type=int)
+@click.option("--cold-start-strategy", default="drop")
+def train_als(ratings_data, split_prop, max_iter, reg_param, rank, cold_start_strategy):
+    seed = 42
+
+    with pyspark.sql.SparkSession.builder.getOrCreate() as spark:
+        ratings_df = spark.read.parquet(ratings_data)
+        (training_df, test_df) = ratings_df.randomSplit([split_prop, 1 - split_prop], seed=seed)
+        training_df.cache()
+        test_df.cache()
+
+        mlflow.log_metric("training_nrows", training_df.count())
+        mlflow.log_metric("test_nrows", test_df.count())
+
+        print(f"Training: {training_df.count()}, test: {test_df.count()}")
+
+        als = (
+            ALS()
+            .setUserCol("userId")
+            .setItemCol("movieId")
+            .setRatingCol("rating")
+            .setPredictionCol("predictions")
+            .setMaxIter(max_iter)
+            .setSeed(seed)
+            .setRegParam(reg_param)
+            .setColdStartStrategy(cold_start_strategy)
+            .setRank(rank)
+        )
+
+        als_model = Pipeline(stages=[als]).fit(training_df)
+
+        reg_eval = RegressionEvaluator(
+            predictionCol="predictions", labelCol="rating", metricName="mse"
+        )
+
+        predicted_test_dF = als_model.transform(test_df)
+
+        test_mse = reg_eval.evaluate(predicted_test_dF)
+        train_mse = reg_eval.evaluate(als_model.transform(training_df))
+
+        print(f"The model had a MSE on the test set of {test_mse}")
+        print(f"The model had a MSE on the (train) set of {train_mse}")
+        mlflow.log_metric("test_mse", test_mse)
+        mlflow.log_metric("train_mse", train_mse)
+        mlflow.spark.log_model(als_model, "als-model")
+
+
+if __name__ == "__main__":
+    train_als()

experimentation_mlops/example/etl_data.py

@@ -0,0 +1,42 @@
+"""
+Converts the raw CSV form to a Parquet form with just the columns we want
+"""
+import os
+import tempfile
+
+import click
+import pyspark
+
+import mlflow
+
+
+@click.command(
+    help="Given a CSV file (see load_raw_data), transforms it into Parquet "
+    "in an mlflow artifact called 'ratings-parquet-dir'"
+)
+@click.option("--ratings-csv")
+@click.option(
+    "--max-row-limit", default=10000, help="Limit the data size to run comfortably on a laptop."
+)
+def etl_data(ratings_csv, max_row_limit):
+    with mlflow.start_run():
+        tmpdir = tempfile.mkdtemp()
+        ratings_parquet_dir = os.path.join(tmpdir, "ratings-parquet")
+        print(f"Converting ratings CSV {ratings_csv} to Parquet {ratings_parquet_dir}")
+        with pyspark.sql.SparkSession.builder.getOrCreate() as spark:
+            ratings_df = (
+                spark.read.option("header", "true")
+                .option("inferSchema", "true")
+                .csv(ratings_csv)
+                .drop("timestamp")
+            )  # Drop unused column
+            ratings_df.show()
+            if max_row_limit != -1:
+                ratings_df = ratings_df.limit(max_row_limit)
+            ratings_df.write.parquet(ratings_parquet_dir)
+            print(f"Uploading Parquet ratings: {ratings_parquet_dir}")
+            mlflow.log_artifacts(ratings_parquet_dir, "ratings-parquet-dir")
+
+
+if __name__ == "__main__":
+    etl_data()

experimentation_mlops/example/load_raw_data.py

@@ -0,0 +1,42 @@
+"""
+Downloads the MovieLens dataset and saves it as an artifact
+"""
+import os
+import tempfile
+import zipfile
+
+import click
+import requests
+
+import mlflow
+
+
+@click.command(
+    help="Downloads the MovieLens dataset and saves it as an mlflow artifact "
+    "called 'ratings-csv-dir'."
+)
+@click.option("--url", default="http://files.grouplens.org/datasets/movielens/ml-20m.zip")
+def load_raw_data(url):
+    with mlflow.start_run():
+        local_dir = tempfile.mkdtemp()
+        local_filename = os.path.join(local_dir, "ml-20m.zip")
+        print(f"Downloading {url} to {local_filename}")
+        r = requests.get(url, stream=True)
+        with open(local_filename, "wb") as f:
+            for chunk in r.iter_content(chunk_size=1024):
+                if chunk:  # filter out keep-alive new chunks
+                    f.write(chunk)
+
+        extracted_dir = os.path.join(local_dir, "ml-20m")
+        print(f"Extracting {local_filename} into {extracted_dir}")
+        with zipfile.ZipFile(local_filename, "r") as zip_ref:
+            zip_ref.extractall(local_dir)
+
+        ratings_file = os.path.join(extracted_dir, "ratings.csv")
+
+        print(f"Uploading ratings: {ratings_file}")
+        mlflow.log_artifact(ratings_file, "ratings-csv-dir")
+
+
+if __name__ == "__main__":
+    load_raw_data()

experimentation_mlops/example/main.py

@@ -0,0 +1,107 @@
+"""
+Downloads the MovieLens dataset, ETLs it into Parquet, trains an
+ALS model, and uses the ALS model to train a Keras neural network.
+
+See README.rst for more details.
+"""
+
+import os
+
+import click
+
+import mlflow
+from mlflow.entities import RunStatus
+from mlflow.tracking import MlflowClient
+from mlflow.tracking.fluent import _get_experiment_id
+from mlflow.utils import mlflow_tags
+from mlflow.utils.logging_utils import eprint
+
+
+def _already_ran(entry_point_name, parameters, git_commit, experiment_id=None):
+    """Best-effort detection of if a run with the given entrypoint name,
+    parameters, and experiment id already ran. The run must have completed
+    successfully and have at least the parameters provided.
+    """
+    experiment_id = experiment_id if experiment_id is not None else _get_experiment_id()
+    client = MlflowClient()
+    all_runs = reversed(client.search_runs([experiment_id]))
+    for run in all_runs:
+        tags = run.data.tags
+        if tags.get(mlflow_tags.MLFLOW_PROJECT_ENTRY_POINT, None) != entry_point_name:
+            continue
+        match_failed = False
+        for param_key, param_value in parameters.items():
+            run_value = run.data.params.get(param_key)
+            if run_value != param_value:
+                match_failed = True
+                break
+        if match_failed:
+            continue
+
+        if run.info.to_proto().status != RunStatus.FINISHED:
+            eprint(
+                ("Run matched, but is not FINISHED, so skipping (run_id={}, status={})").format(
+                    run.info.run_id, run.info.status
+                )
+            )
+            continue
+
+        previous_version = tags.get(mlflow_tags.MLFLOW_GIT_COMMIT, None)
+        if git_commit != previous_version:
+            eprint(
+                "Run matched, but has a different source version, so skipping "
+                f"(found={previous_version}, expected={git_commit})"
+            )
+            continue
+        return client.get_run(run.info.run_id)
+    eprint("No matching run has been found.")
+    return None
+
+
+# TODO(aaron): This is not great because it doesn't account for:
+# - changes in code
+# - changes in dependent steps
+def _get_or_run(entrypoint, parameters, git_commit, use_cache=True):
+    existing_run = _already_ran(entrypoint, parameters, git_commit)
+    if use_cache and existing_run:
+        print(f"Found existing run for entrypoint={entrypoint} and parameters={parameters}")
+        return existing_run
+    print(f"Launching new run for entrypoint={entrypoint} and parameters={parameters}")
+    submitted_run = mlflow.run(".", entrypoint, parameters=parameters, env_manager="local")
+    return MlflowClient().get_run(submitted_run.run_id)
+
+
+@click.command()
+@click.option("--als-max-iter", default=10, type=int)
+@click.option("--keras-hidden-units", default=20, type=int)
+@click.option("--max-row-limit", default=100000, type=int)
+def workflow(als_max_iter, keras_hidden_units, max_row_limit):
+    # Note: The entrypoint names are defined in MLproject. The artifact directories
+    # are documented by each step's .py file.
+    with mlflow.start_run() as active_run:
+        os.environ["SPARK_CONF_DIR"] = os.path.abspath(".")
+        git_commit = active_run.data.tags.get(mlflow_tags.MLFLOW_GIT_COMMIT)
+        load_raw_data_run = _get_or_run("load_raw_data", {}, git_commit)
+        ratings_csv_uri = os.path.join(load_raw_data_run.info.artifact_uri, "ratings-csv-dir")
+        etl_data_run = _get_or_run(
+            "etl_data", {"ratings_csv": ratings_csv_uri, "max_row_limit": max_row_limit}, git_commit
+        )
+        ratings_parquet_uri = os.path.join(etl_data_run.info.artifact_uri, "ratings-parquet-dir")
+
+        # We specify a spark-defaults.conf to override the default driver memory. ALS requires
+        # significant memory. The driver memory property cannot be set by the application itself.
+        als_run = _get_or_run(
+            "als", {"ratings_data": ratings_parquet_uri, "max_iter": str(als_max_iter)}, git_commit
+        )
+        als_model_uri = os.path.join(als_run.info.artifact_uri, "als-model")
+
+        keras_params = {
+            "ratings_data": ratings_parquet_uri,
+            "als_model_uri": als_model_uri,
+            "hidden_units": keras_hidden_units,
+        }
+        _get_or_run("train_keras", keras_params, git_commit, use_cache=False)
+
+
+if __name__ == "__main__":
+    workflow()

experimentation_mlops/example/python_env.yaml

@@ -0,0 +1,10 @@
+python: "3.8"
+build_dependencies:
+  - pip
+dependencies:
+  - tensorflow==1.15.2
+  - keras==2.2.4
+  - mlflow>=1.0
+  - pyspark
+  - requests
+  - click

experimentation_mlops/example/spark-defaults.conf

@@ -0,0 +1 @@
+spark.driver.memory 8g

experimentation_mlops/example/train_keras.py

@@ -0,0 +1,116 @@
+"""
+Trains a Keras model for user/movie ratings. The input is a Parquet
+ratings dataset (see etl_data.py) and an ALS model (see als.py), which we
+will use to supplement our input and train using.
+"""
+from itertools import chain
+
+import click
+import numpy as np
+import pandas as pd
+import pyspark
+import tensorflow as tf
+from pyspark.sql.functions import col, udf
+from pyspark.sql.types import ArrayType, FloatType
+from tensorflow import keras
+from tensorflow.keras.callbacks import EarlyStopping
+from tensorflow.keras.layers import Dense
+from tensorflow.keras.models import Sequential
+
+import mlflow
+import mlflow.spark
+
+
+@click.command()
+@click.option("--ratings-data", help="Path readable by Spark to the ratings Parquet file")
+@click.option("--als-model-uri", help="Path readable by load_model to ALS MLmodel")
+@click.option("--hidden-units", default=20, type=int)
+def train_keras(ratings_data, als_model_uri, hidden_units):
+    np.random.seed(0)
+    tf.set_random_seed(42)  # For reproducibility
+
+    with pyspark.sql.SparkSession.builder.getOrCreate() as spark:
+        als_model = mlflow.spark.load_model(als_model_uri).stages[0]
+        ratings_df = spark.read.parquet(ratings_data)
+        (training_df, test_df) = ratings_df.randomSplit([0.8, 0.2], seed=42)
+        training_df.cache()
+        test_df.cache()
+
+        mlflow.log_metric("training_nrows", training_df.count())
+        mlflow.log_metric("test_nrows", test_df.count())
+
+        print(f"Training: {training_df.count()}, test: {test_df.count()}")
+
+        user_factors = als_model.userFactors.selectExpr("id as userId", "features as uFeatures")
+        item_factors = als_model.itemFactors.selectExpr("id as movieId", "features as iFeatures")
+        joined_train_df = training_df.join(item_factors, on="movieId").join(
+            user_factors, on="userId"
+        )
+        joined_test_df = test_df.join(item_factors, on="movieId").join(user_factors, on="userId")
+
+        # We'll combine the movies and ratings vectors into a single vector of length 24.
+        # We will then explode this features vector into a set of columns.
+        def concat_arrays(*args):
+            return list(chain(*args))
+
+        concat_arrays_udf = udf(concat_arrays, ArrayType(FloatType()))
+
+        concat_train_df = joined_train_df.select(
+            "userId",
+            "movieId",
+            concat_arrays_udf(col("iFeatures"), col("uFeatures")).alias("features"),
+            col("rating").cast("float"),
+        )
+        concat_test_df = joined_test_df.select(
+            "userId",
+            "movieId",
+            concat_arrays_udf(col("iFeatures"), col("uFeatures")).alias("features"),
+            col("rating").cast("float"),
+        )
+
+        pandas_df = concat_train_df.toPandas()
+        pandas_test_df = concat_test_df.toPandas()
+
+        # This syntax will create a new DataFrame where elements of the 'features' vector
+        # are each in their own column. This is what we'll train our neural network on.
+        x_test = pd.DataFrame(pandas_test_df.features.values.tolist(), index=pandas_test_df.index)
+        x_train = pd.DataFrame(pandas_df.features.values.tolist(), index=pandas_df.index)
+
+        # Show matrix for example.
+        print("Training matrix:")
+        print(x_train)
+
+        # Create our Keras model with two fully connected hidden layers.
+        model = Sequential()
+        model.add(Dense(30, input_dim=24, activation="relu"))
+        model.add(Dense(hidden_units, activation="relu"))
+        model.add(Dense(1, activation="linear"))
+
+        model.compile(loss="mse", optimizer=keras.optimizers.Adam(lr=0.0001))
+
+        early_stopping = EarlyStopping(
+            monitor="val_loss", min_delta=0.0001, patience=2, mode="auto"
+        )
+
+        model.fit(
+            x_train,
+            pandas_df["rating"],
+            validation_split=0.2,
+            verbose=2,
+            epochs=3,
+            batch_size=128,
+            shuffle=False,
+            callbacks=[early_stopping],
+        )
+
+        train_mse = model.evaluate(x_train, pandas_df["rating"], verbose=2)
+        test_mse = model.evaluate(x_test, pandas_test_df["rating"], verbose=2)
+        mlflow.log_metric("test_mse", test_mse)
+        mlflow.log_metric("train_mse", train_mse)
+
+        print(f"The model had a MSE on the test set of {test_mse}")
+        mlflow.tensorflow.log_model(model, "keras-model")
+
+
+if __name__ == "__main__":
+    train_keras()

experimentation_mlops/mlops/MLProject

@@ -0,0 +1,13 @@
+name: multistep_example
+
+python_env: python_env.yaml
+
+entry_points:
+ ingest_request:
+  command: "python ingest_request.py"
+
+ ingest_convert:
+  command: "python ingest_convert.py"
+
+ main:
+  command: "python main.py"

experimentation_mlops/mlops/data/2week_news_data.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:17542732c79d799aa9bcc1b8f7fbf6e11da1fc51c1c21088ebf7572f6d922862
+size 843591

experimentation_mlops/mlops/data/2week_news_data.zip

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:64638c216b8479c03046114e5971943d8be624f025fdf827c18c46806743d922
+size 520334

experimentation_mlops/mlops/desktop.ini

@@ -0,0 +1,4 @@
+[ViewState]
+Mode=
+Vid=
+FolderType=Generic

experimentation_mlops/mlops/evaluation.py

@@ -0,0 +1,42 @@
+## TODO
+
+## Accepts a dataset name and model id, predict and produces graph + error metrics
+
+
+import matplotlib.pyplot as plt
+import mlflow
+import mlflow.data
+import numpy as np
+import pandas as pd
+from mlflow.client import MlflowClient
+from mlflow.data.pandas_dataset import PandasDataset
+from utilsforecast.plotting import plot_series
+
+from neuralforecast.core import NeuralForecast
+from neuralforecast.models import NBEATSx
+from neuralforecast.utils import AirPassengersDF
+from neuralforecast.losses.pytorch import MAE
+
+import matplotlib.pyplot as plt
+Y_plot = Y_hat_df[Y_hat_df['unique_id']=='Dated']
+cutoffs = Y_hat_df['cutoff'].unique()[::horizon]
+Y_plot = Y_plot[Y_hat_df['cutoff'].isin(cutoffs)]
+
+plt.figure(figsize=(20,5))
+plt.plot(Y_plot['ds'], Y_plot['y'], label='True')
+for model in models:
+    plt.plot(Y_plot['ds'], Y_plot[f'{model}'], label=f'{model}')
+plt.xlabel('Datestamp')
+plt.ylabel('OT')
+plt.grid()
+plt.legend()
+
+from neuralforecast.losses.numpy import mse, mae, mape
+
+for model in models:
+    mae_model = mae(Y_hat_df['y'], Y_hat_df[f'{model}'])
+    mse_model = mse(Y_hat_df['y'], Y_hat_df[f'{model}'])
+    mape_model = mape(Y_hat_df['y'], Y_hat_df[f'{model}'])
+    print(f'{model} horizon {horizon} - MAE: {mae_model:.3f}')
+    print(f'{model} horizon {horizon} - MSE: {mse_model:.3f}')
+    print(f'{model} horizon {horizon} - MAPE: {mape_model:.3f}')

experimentation_mlops/mlops/ingest_convert.py

@@ -0,0 +1,51 @@
+"""
+Converts the raw CSV form to a Parquet form with just the columns we want
+"""
+import os
+import tempfile
+
+import click
+import pandas as pd
+
+import mlflow
+
+
+@click.command(
+    help="Given a CSV file (see load_raw_data), transforms it into Parquet "
+    "in an mlflow artifact called 'data-parquet-dir'"
+)
+@click.option("--data-csv")
+@click.option(
+    "--max-row-limit", default=10000, help="Limit the data size to run comfortably on a laptop."
+)
+def ingest_convert(data_csv, max_row_limit):
+    with mlflow.start_run():
+        tmpdir = tempfile.mkdtemp()
+        data_parquet_dir = os.path.join(tmpdir, "data-parquet")
+        print(f"Converting data CSV {data_csv} to Parquet {data_parquet_dir}")
+
+        # data_csv = fr"{data_csv}"
+        # print(data_csv)
+        # dirName = data_csv.replace("file:///", "")
+
+        # fn = [f for f in os.listdir(dirName)\
+        #     if f.endswith('.csv') and os.path.isfile(os.path.join(dirName, f))][0]
+        # data_csv_file = os.path.join(dirName, fn)
+
+        data_df = pd.read_csv(data_csv)
+        data_df.to_parquet(data_parquet_dir)
+
+        # table = pa.csv.read_csv(data_csv)
+        # pa.parquet.write_table(table, data_csv.replace('csv', 'parquet'))
+        
+        if max_row_limit != -1:
+            data_df = data_df.iloc[:max_row_limit]
+
+        # data_df.write.parquet(data_parquet_dir)
+
+        print(f"Uploading Parquet data: {data_parquet_dir}")
+        mlflow.log_artifacts(data_parquet_dir, "data-parquet-dir")
+
+
+if __name__ == "__main__":
+    ingest_convert()

experimentation_mlops/mlops/ingest_request.py

@@ -0,0 +1,54 @@
+"""
+This module defines the following routines used by the 'ingest' step of the time series forecasting flow:
+
+- ``load_file_as_dataframe``: Defines customizable logic for parsing dataset formats that are not
+  natively parsed by MLflow Recipes (i.e. formats other than Parquet, Delta, and Spark SQL).
+"""
+
+import pandas as pd
+import os
+import tempfile
+import click
+import mlflow
+
+import gdown
+import requests
+import zipfile
+
+@click.command(
+    help="Downloads the dataset and saves it as an mlflow artifact "
+    "called 'data-csv-dir'."
+)
+@click.option("--url", default="https://drive.google.com/uc?id=1H8RHsrgYMd6VC23_OJqrN6o_mL78pWpx")
+def ingest_request(url) -> pd.DataFrame:
+    """
+    Downloads data from the specified url.
+
+    :param url: Url to the dataset file.
+    :return: MLflow artifact containing the downloaded data in its raw form.
+    """
+    with mlflow.start_run():
+        local_dir = tempfile.mkdtemp()
+        local_filename = os.path.join(local_dir, "news-data.zip")
+        print(f"Downloading {url} to {local_filename}")
+        # r = requests.get(url, stream=True)
+        # with open(local_filename, "wb") as f:
+        #     for chunk in r.iter_content(chunk_size=1024):
+        #         if chunk:  # filter out keep-alive new chunks
+        #             f.write(chunk)
+
+        gdown.download(url, local_filename, quiet=False)
+        extracted_dir = os.path.join(local_dir)
+        print(f"Extracting {local_filename} into {extracted_dir}")
+
+        with zipfile.ZipFile(local_filename, "r") as zip_ref:
+            zip_ref.extractall(local_dir)
+
+        data_file = os.path.join(extracted_dir, "2week_news_data.csv")
+
+        print(f"Uploading data: {data_file}")
+        mlflow.log_artifact(data_file, "data-csv-dir")
+    
+if __name__ == "__main__":
+    ingest_request()
+    

experimentation_mlops/mlops/main.py

@@ -0,0 +1,104 @@
+"""
+Time series forecasting
+"""
+
+import os
+
+import click
+
+import mlflow
+from mlflow.entities import RunStatus
+from mlflow.tracking import MlflowClient
+from mlflow.tracking.fluent import _get_experiment_id
+from mlflow.utils import mlflow_tags
+from mlflow.utils.logging_utils import eprint
+
+
+def _already_ran(entry_point_name, parameters, git_commit, experiment_id=None):
+    """Best-effort detection of if a run with the given entrypoint name,
+    parameters, and experiment id already ran. The run must have completed
+    successfully and have at least the parameters provided.
+    """
+    experiment_id = experiment_id if experiment_id is not None else _get_experiment_id()
+    client = MlflowClient()
+    all_runs = reversed(client.search_runs([experiment_id]))
+    for run in all_runs:
+        tags = run.data.tags
+        if tags.get(mlflow_tags.MLFLOW_PROJECT_ENTRY_POINT, None) != entry_point_name:
+            continue
+        match_failed = False
+        for param_key, param_value in parameters.items():
+            run_value = run.data.params.get(param_key)
+            if run_value != param_value:
+                match_failed = True
+                break
+        if match_failed:
+            continue
+
+        if run.info.to_proto().status != RunStatus.FINISHED:
+            eprint(
+                ("Run matched, but is not FINISHED, so skipping (run_id={}, status={})").format(
+                    run.info.run_id, run.info.status
+                )
+            )
+            continue
+
+        previous_version = tags.get(mlflow_tags.MLFLOW_GIT_COMMIT, None)
+        if git_commit != previous_version:
+            eprint(
+                "Run matched, but has a different source version, so skipping "
+                f"(found={previous_version}, expected={git_commit})"
+            )
+            continue
+        return client.get_run(run.info.run_id)
+    eprint("No matching run has been found.")
+    return None
+
+
+# TODO(aaron): This is not great because it doesn't account for:
+# - changes in code
+# - changes in dependent steps
+def _get_or_run(entrypoint, parameters, git_commit, use_cache=True):
+    existing_run = _already_ran(entrypoint, parameters, git_commit)
+    if use_cache and existing_run:
+        print(f"Found existing run for entrypoint={entrypoint} and parameters={parameters}")
+        return existing_run
+    print(f"Launching new run for entrypoint={entrypoint} and parameters={parameters}")
+    submitted_run = mlflow.run(".", entrypoint, parameters=parameters, env_manager="local")
+    return MlflowClient().get_run(submitted_run.run_id)
+
+
+@click.command()
+@click.option("--max-row-limit", default=100000, type=int)
+
+def workflow(max_row_limit):
+    # Note: The entrypoint names are defined in MLproject. The artifact directories
+    # are documented by each step's .py file.
+    with mlflow.start_run() as active_run:
+        os.environ["SPARK_CONF_DIR"] = os.path.abspath(".")
+        git_commit = active_run.data.tags.get(mlflow_tags.MLFLOW_GIT_COMMIT)
+        ingest_request_run = _get_or_run("ingest_request", {}, git_commit)
+        data_csv_uri = os.path.join(ingest_request_run.info.artifact_uri, "data-csv-dir")
+        print(data_csv_uri)
+        ingest_convert_run = _get_or_run(
+            "ingest_convert", {"data-csv": data_csv_uri, "max-row-limit": max_row_limit}, git_commit
+        )
+        data_parquet_uri = os.path.join(ingest_convert_run.info.artifact_uri, "data-parquet-dir")
+
+        # We specify a spark-defaults.conf to override the default driver memory. ALS requires
+        # significant memory. The driver memory property cannot be set by the application itself.
+        # als_run = _get_or_run(
+        #     "als", {"ratings_data": ratings_parquet_uri, "max_iter": str(als_max_iter)}, git_commit
+        # )
+        # als_model_uri = os.path.join(als_run.info.artifact_uri, "als-model")
+
+        # keras_params = {
+        #     "ratings_data": ratings_parquet_uri,
+        #     "als_model_uri": als_model_uri,
+        #     "hidden_units": keras_hidden_units,
+        # }
+        # _get_or_run("train_keras", keras_params, git_commit, use_cache=False)
+
+
+if __name__ == "__main__":
+    workflow()

experimentation_mlops/mlops/ml-doc.md

@@ -0,0 +1,59 @@
+# Machine Learning Operations (MLOps) Pipeline Documentation
+
+This is the documentation covering  each of the steps included in Bioma AI's time-series-forecasting MLOps Pipeline.
+
+## Sequential MLOps Steps
+The information flow of the pipeline will closely resemble that of a regression machine learning task. The model development will consist of sequential steps:
+1. Ingestion,
+2. Transformation,
+3. Training,
+4. Evaluation, and
+5. Registration.
+
+![](pics/pipeline.png)MLFlow Regression Recipe's Information Flow [1]
+
+## 1. Ingestion
+
+Our pipeline involves extracting raw datasets from the internet (S3 Buckets and other cloud services), the assumed dataset is of one of the following file types: csv, json, parquet or xlsx. The extracted data is saved as an artifact which can help in documentation purposes.
+
+In the case of time series forecasting, the data ingestion process is tasked on receiving data from a specific format and converting it to a Pandas Dataframe for further processing. The data will be downloaded from the web by issuing a request, the data will then be converted into parquet before being written as a Pandas dataframe. The parquet file will be saved as an artifact for the purpose of documentation.
+
+## 2. Transformation
+
+According to the timeframe of the time-series data, the data will be split into a train-test-validation set. The user will be able to customize each of the set's proportions.
+
+Various statistical methods is considered and performed into a selection of columns, the columns and the methods are both customizable. A few methods that are considered are:
+1. Logarithmic
+2. Natural Logarithmic
+3. Standardization
+4. Identity
+5. Logarithmic Difference
+
+## 3. Training
+
+The training process can be broken down into two types according to the amount of variates being predicted: univariate or multivariate.
+
+Predictors are either an:
+
+1. Endogenous feature (Changes in the target's value has an effect on the predictor's value or the other way around) or 
+2. Exogenous feature (changes in the predictor's value has an effect on the target's value, but not the other way around)
+    <ol type="a">
+    <li>Static Exogenous</li>
+    Static variables such as one-hot encoding for a categorical class identifier.
+    <li>Historical Exogenous</li>
+    Exogenous features that their historical data is only known of.
+    <li>Future Exogenous</li>
+    Exogenous features that their data is known of when making the prediction on that time in the future.
+    </ol>
+
+Endogenous features will be predicted in conjunction with the target's feature. Exogenous features will not be predicted, rather only be used to predict the target variable.
+
+In short: multivariate predictions will use predictors as endogenous features, while multivariable predictions use predictors as exogenous features because of its univariate nature.
+
+## 4. Evaluation
+
+The evaluation step is constructed for the trained models to perform prediction on out-of-training data. Ideally, this step will produce outputs such as visualizations and error metrics for arbitrary datasets. 
+
+References:
+- [1] [mlflow/recipes-regression-template](https://github.com/mlflow/recipes-regression-template/tree/main?tab=readme-ov-file#installation)
+- [2] [MLflow deployment using Docker, EC2, S3, and RDS](https://aws.plainenglish.io/set-up-mlflow-on-aws-ec2-using-docker-s3-and-rds-90d96798e555)

experimentation_mlops/mlops/modules/transformations.py

@@ -0,0 +1,39 @@
+"""
+Stores all transformations
+"""
+def createLag(data, amt=10):
+    """
+    Create a lag inside dataframe, in business days
+
+    :param pandas.DataFrame data:
+    :param int amt: Unit of lag period
+
+    :return: Copy of pandas Dataframe
+    """
+    import pandas as pd
+    if 'ds' in data:
+        copy = data.copy()
+        copy['ds'] = copy['ds'] + pd.tseries.offsets.BusinessDay(amt)
+        return copy
+    else:
+        print(f"No 'ds' column found inside dataframe")
+        return data
+    
+def scaleStandard(df_col):
+    """
+    Fits and returns a standard scaled version of a dataframe column
+    """
+    import pandas as pd
+    from sklearn.preprocessing import StandardScaler
+    scaler = StandardScaler()
+    df_col = scaler.fit_transform(df_col)
+    df_col = pd.DataFrame(df_col)
+    return df_col, scaler
+
+def logReturn(data, df_col):
+    """
+    Perform log return for a dataframe column
+    """
+    import numpy as np
+    new_col = np.log1p(data[df_col].pct_change())
+    return new_col

experimentation_mlops/mlops/python_env.yaml

@@ -0,0 +1,11 @@
+python: "3.9"
+build_dependencies:
+ - pip
+dependencies:
+ - pytorch
+ - openpyxl
+ - pandas
+ - mlflow
+ - pyspark
+ - requests
+ - click

experimentation_mlops/mlops/requirements.txt

@@ -0,0 +1,32 @@
+numpy
+scipy
+scikit-learn
+fastapi
+ipykernel
+flask
+beautifulsoup4
+--extra-index-url https://download.pytorch.org/whl/cu118
+torch
+torchvision
+torchaudio
+mlflow
+pandas
+transformers
+fsspec
+gitpython
+hyperopt
+jupyterlab
+matplotlib
+numba
+numpy
+optuna
+pyarrow
+pytorch-lightning
+pip
+s3fs
+nbdev
+black
+polars
+ray[tune]>=2.2.0
+utilsforecast>=0.0.24
+coreforecast

experimentation_mlops/mlops/spark-defaults.conf

@@ -0,0 +1 @@
+spark.driver.memory 8g

experimentation_mlops/mlops/test.ipynb

@@ -0,0 +1,490 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Downloading...\n",
+      "From: https://drive.google.com/uc?id=1H8RHsrgYMd6VC23_OJqrN6o_mL78pWpx\n",
+      "To: e:\\projects\\Bioma-AI\\1-time-series-forecasting\\trend_prediction_app\\trend-prediction-ml\\trend-prediction-pipeline\\collected-news.zip\n",
+      "100%|██████████| 520k/520k [00:02<00:00, 241kB/s]\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "'collected-news.zip'"
+      ]
+     },
+     "execution_count": 1,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "import gdown\n",
+    "\n",
+    "url = 'https://drive.google.com/uc?id=1H8RHsrgYMd6VC23_OJqrN6o_mL78pWpx'\n",
+    "output = 'collected-news.zip'\n",
+    "gdown.download(url, output, quiet=False)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pyarrow.csv as pv\n",
+    "import pyarrow.parquet as pq\n",
+    "\n",
+    "filename = 'data/2week_news_data.csv'\n",
+    "\n",
+    "table = pv.read_csv(filename)\n",
+    "pq.write_table(table, filename.replace('csv', 'parquet'))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "'E:\\\\projects\\\\Bioma-AI\\\\time-series-forecasting\\\\trend_prediction_app\\\\trend-prediction-ml\\\\trend-prediction-pipeline\\\\data\\\\2week_news_data.csv'"
+      ]
+     },
+     "execution_count": 3,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "import os\n",
+    "\n",
+    "dirName = r\"E:\\projects\\Bioma-AI\\time-series-forecasting\\trend_prediction_app\\trend-prediction-ml\\trend-prediction-pipeline\\data\"\n",
+    "fn = [f for f in os.listdir(dirName)\\\n",
+    "    if f.endswith('.csv') and os.path.isfile(os.path.join(dirName, f))][0]\n",
+    "path = os.path.join(dirName, fn)\n",
+    "path"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "'E:\\\\projects\\\\Bioma-AI\\x01-time-series-forecasting\\trend_prediction_app\\trend-prediction-ml\\trend-prediction-pipeline\\\\data'"
+      ]
+     },
+     "execution_count": 1,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "def to_raw(string):\n",
+    "    return fr\"{string}\"\n",
+    "\n",
+    "dirName = to_raw(\"E:\\projects\\Bioma-AI\\1-time-series-forecasting\\trend_prediction_app\\trend-prediction-ml\\trend-prediction-pipeline\\data\")\n",
+    "\n",
+    "dirName = dirName.replace(\"file:///\", \"\")\n",
+    "\n",
+    "dirName\n",
+    "\n",
+    "dirName = rf\"{dirName}\"\n",
+    "\n",
+    "dirName"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "'E:\\\\projects\\\\Bioma-AI\\\\time-series-forecasting\\\\trend_prediction_app\\\\trend-prediction-ml\\\\trend-prediction-pipeline\\\\data\\\\2week_news_data.csv'"
+      ]
+     },
+     "execution_count": 6,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "dirName = r\"file:///E:\\projects\\Bioma-AI\\time-series-forecasting\\trend_prediction_app\\trend-prediction-ml\\trend-prediction-pipeline\\data\"\n",
+    "\n",
+    "dirName = dirName.replace(\"file:///\", \"\")\n",
+    "\n",
+    "fn = [f for f in os.listdir(dirName)\\\n",
+    "    if f.endswith('.csv') and os.path.isfile(os.path.join(dirName, f))][0]\n",
+    "path = os.path.join(dirName, fn)\n",
+    "path"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "        # r = requests.get(url, stream=True)\n",
+    "        # with open(local_filename, \"wb\") as f:\n",
+    "        #     for chunk in r.iter_content(chunk_size=1024):\n",
+    "        #         if chunk:  # filter out keep-alive new chunks\n",
+    "        #             f.write(chunk)\n",
+    "\n",
+    "        # data_csv = fr\"{data_csv}\"\n",
+    "        # print(data_csv)\n",
+    "        # dirName = data_csv.replace(\"file:///\", \"\")\n",
+    "\n",
+    "        # fn = [f for f in os.listdir(dirName)\\\n",
+    "        #     if f.endswith('.csv') and os.path.isfile(os.path.join(dirName, f))][0]\n",
+    "        # data_csv_file = os.path.join(dirName, fn)\n",
+    "\n",
+    "        # table = pa.csv.read_csv(data_csv)\n",
+    "        # pa.parquet.write_table(table, data_csv.replace('csv', 'parquet'))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## SQLite"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "## Delete Data\n",
+    "\n",
+    "import sqlite3\n",
+    "\n",
+    "try:\n",
+    "    with sqlite3.connect('my.db') as conn:\n",
+    "        cur = conn.cursor()\n",
+    "        delete_stmt = 'DELETE FROM mlruns.db WHERE id = ?'\n",
+    "        cur.execute(delete_stmt, (1,))\n",
+    "        conn.commit()\n",
+    "except sqlite3.Error as e:\n",
+    "    print(e)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "no such table: mlruns.db\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Select Data\n",
+    "\n",
+    "import sqlite3\n",
+    "\n",
+    "try:\n",
+    "    with sqlite3.connect(r'E:\\Projects\\Bioma-AI\\time-series-forecasting\\trend_prediction_app\\trend-prediction-ml\\trend-prediction-pipeline\\mlruns.db') as conn:\n",
+    "        cur = conn.cursor()\n",
+    "        delete_stmt = 'SELECT * FROM mlruns.db'\n",
+    "        cur.execute(delete_stmt, (1,))\n",
+    "        conn.commit()\n",
+    "except sqlite3.Error as e:\n",
+    "    print(e)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "('e1dd6f53468a43ab8ae216ecd6d00f9b', 'fun-pug-865', 'UNKNOWN', '', '', 'mryan', 'FINISHED', 1725156717665, 1725156717711, '', 'deleted', 'file:///E:/Projects/Bioma-AI/time-series-forecasting/trend_prediction_app/trend-prediction-ml/trend-prediction-pipeline/mlruns/0/e1dd6f53468a43ab8ae216ecd6d00f9b/artifacts', 0, 1725157415352)\n",
+      "('19b0aa8dfd4d43babf1722241eac4d11', 'amazing-flea-532', 'UNKNOWN', '', '', 'mryan', 'FINISHED', 1725156844490, 1725156844527, '', 'deleted', 'file:///E:/Projects/Bioma-AI/time-series-forecasting/trend_prediction_app/trend-prediction-ml/trend-prediction-pipeline/mlruns/0/19b0aa8dfd4d43babf1722241eac4d11/artifacts', 0, 1725157415338)\n",
+      "('e8fb896431de4e37a188ae039901a502', 'rumbling-penguin-621', 'UNKNOWN', '', '', 'mryan', 'FINISHED', 1725157265761, 1725157265802, '', 'deleted', 'file:///E:/Projects/Bioma-AI/time-series-forecasting/trend_prediction_app/trend-prediction-ml/trend-prediction-pipeline/mlruns/0/e8fb896431de4e37a188ae039901a502/artifacts', 0, 1725157415321)\n",
+      "('4efbf5256ae34e6296a00238accfdc9f', 'trusting-doe-440', 'UNKNOWN', '', '', 'mryan', 'FINISHED', 1725157268161, 1725157268196, '', 'deleted', 'file:///E:/Projects/Bioma-AI/time-series-forecasting/trend_prediction_app/trend-prediction-ml/trend-prediction-pipeline/mlruns/0/4efbf5256ae34e6296a00238accfdc9f/artifacts', 0, 1725157415324)\n",
+      "('b0c0116c1805431a953bb8c07c184de9', 'popular-slug-621', 'UNKNOWN', '', '', 'mryan', 'FINISHED', 1725157437890, 1725157437931, '', 'active', 'file:///E:/Projects/Bioma-AI/time-series-forecasting/trend_prediction_app/trend-prediction-ml/trend-prediction-pipeline/mlruns/0/b0c0116c1805431a953bb8c07c184de9/artifacts', 0, None)\n"
+     ]
+    }
+   ],
+   "source": [
+    "import mlflow\n",
+    "import sqlite3\n",
+    "\n",
+    "# Log some data with MLflow\n",
+    "mlflow.set_tracking_uri(\"sqlite:///mlruns.db\")\n",
+    "\n",
+    "with mlflow.start_run():\n",
+    "    mlflow.log_param(\"param1\", 5)\n",
+    "    mlflow.log_metric(\"metric1\", 0.1)\n",
+    "\n",
+    "# Query the SQLite database\n",
+    "try:\n",
+    "    db_path = r'E:\\Projects\\Bioma-AI\\time-series-forecasting\\trend_prediction_app\\trend-prediction-ml\\trend-prediction-pipeline\\mlruns.db'\n",
+    "    with sqlite3.connect(db_path) as conn:\n",
+    "        cur = conn.cursor()\n",
+    "        \n",
+    "        # Example query to fetch all runs\n",
+    "        select_stmt = 'SELECT * FROM runs'\n",
+    "        cur.execute(select_stmt)\n",
+    "        rows = cur.fetchall()\n",
+    "        \n",
+    "        for row in rows:\n",
+    "            print(row)\n",
+    "\n",
+    "except sqlite3.Error as e:\n",
+    "    print(f\"SQLite error: {e}\")\n",
+    "except Exception as e:\n",
+    "    print(f\"General error: {e}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Tables in database: [('experiments',), ('alembic_version',), ('experiment_tags',), ('tags',), ('registered_models',), ('runs',), ('registered_model_tags',), ('model_version_tags',), ('model_versions',), ('latest_metrics',), ('metrics',), ('registered_model_aliases',), ('datasets',), ('inputs',), ('input_tags',), ('params',), ('trace_info',), ('trace_tags',), ('trace_request_metadata',)]\n"
+     ]
+    }
+   ],
+   "source": [
+    "import sqlite3\n",
+    "\n",
+    "db_path = r\"E:\\Projects\\Bioma-AI\\time-series-forecasting\\trend_prediction_app\\trend-prediction-ml\\trend-prediction-pipeline\\mlruns.db\"\n",
+    "\n",
+    "try:\n",
+    "    conn = sqlite3.connect(db_path)\n",
+    "    cursor = conn.cursor()\n",
+    "    cursor.execute(\"SELECT name FROM sqlite_master WHERE type='table';\")\n",
+    "    tables = cursor.fetchall()\n",
+    "    print(\"Tables in database:\", tables)\n",
+    "except sqlite3.Error as e:\n",
+    "    print(f\"SQLite error: {e}\")\n",
+    "finally:\n",
+    "    conn.close()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Run ID: a930976208fa4fb48dbbf5cf4d7f600f\n"
+     ]
+    }
+   ],
+   "source": [
+    "import mlflow\n",
+    "\n",
+    "# Use a new SQLite database file\n",
+    "new_db_path = r\"E:\\Projects\\Bioma-AI\\time-series-forecasting\\trend_prediction_app\\trend-prediction-ml\\trend-prediction-pipeline\\new_mlruns.db\"\n",
+    "mlflow.set_tracking_uri(f\"sqlite:///{new_db_path}\")\n",
+    "\n",
+    "with mlflow.start_run() as run:\n",
+    "    mlflow.log_param(\"param1\", \"test\")\n",
+    "    print(f\"Run ID: {run.info.run_id}\")\n",
+    "\n",
+    "# Check if the run is properly logged\n",
+    "client = mlflow.tracking.MlflowClient()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {},
+   "outputs": [
+    {
+     "ename": "MlflowException",
+     "evalue": "Run with id=a930976208fa4fb48dbbf5cf4d7f600f not found",
+     "output_type": "error",
+     "traceback": [
+      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
+      "\u001b[1;31mMlflowException\u001b[0m                           Traceback (most recent call last)",
+      "Cell \u001b[1;32mIn[7], line 12\u001b[0m\n\u001b[0;32m     10\u001b[0m \u001b[38;5;66;03m# Get run information using the run ID\u001b[39;00m\n\u001b[0;32m     11\u001b[0m run_id \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124ma930976208fa4fb48dbbf5cf4d7f600f\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m---> 12\u001b[0m run \u001b[38;5;241m=\u001b[39m \u001b[43mclient\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mget_run\u001b[49m\u001b[43m(\u001b[49m\u001b[43mrun_id\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m     13\u001b[0m \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mRun ID: \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mrun\u001b[38;5;241m.\u001b[39minfo\u001b[38;5;241m.\u001b[39mrun_id\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m\"\u001b[39m)\n\u001b[0;32m     14\u001b[0m \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mRun Status: \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mrun\u001b[38;5;241m.\u001b[39minfo\u001b[38;5;241m.\u001b[39mstatus\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m\"\u001b[39m)\n",
+      "File \u001b[1;32me:\\VM\\miniconda3\\envs\\mlops\\lib\\site-packages\\mlflow\\tracking\\client.py:226\u001b[0m, in \u001b[0;36mMlflowClient.get_run\u001b[1;34m(self, run_id)\u001b[0m\n\u001b[0;32m    182\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mget_run\u001b[39m(\u001b[38;5;28mself\u001b[39m, run_id: \u001b[38;5;28mstr\u001b[39m) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m Run:\n\u001b[0;32m    183\u001b[0m \u001b[38;5;250m    \u001b[39m\u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[0;32m    184\u001b[0m \u001b[38;5;124;03m    Fetch the run from backend store. The resulting :py:class:`Run <mlflow.entities.Run>`\u001b[39;00m\n\u001b[0;32m    185\u001b[0m \u001b[38;5;124;03m    contains a collection of run metadata -- :py:class:`RunInfo <mlflow.entities.RunInfo>`,\u001b[39;00m\n\u001b[1;32m   (...)\u001b[0m\n\u001b[0;32m    224\u001b[0m \n\u001b[0;32m    225\u001b[0m \u001b[38;5;124;03m    \"\"\"\u001b[39;00m\n\u001b[1;32m--> 226\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_tracking_client\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mget_run\u001b[49m\u001b[43m(\u001b[49m\u001b[43mrun_id\u001b[49m\u001b[43m)\u001b[49m\n",
+      "File \u001b[1;32me:\\VM\\miniconda3\\envs\\mlops\\lib\\site-packages\\mlflow\\tracking\\_tracking_service\\client.py:104\u001b[0m, in \u001b[0;36mTrackingServiceClient.get_run\u001b[1;34m(self, run_id)\u001b[0m\n\u001b[0;32m     88\u001b[0m \u001b[38;5;250m\u001b[39m\u001b[38;5;124;03m\"\"\"Fetch the run from backend store. The resulting :py:class:`Run <mlflow.entities.Run>`\u001b[39;00m\n\u001b[0;32m     89\u001b[0m \u001b[38;5;124;03mcontains a collection of run metadata -- :py:class:`RunInfo <mlflow.entities.RunInfo>`,\u001b[39;00m\n\u001b[0;32m     90\u001b[0m \u001b[38;5;124;03mas well as a collection of run parameters, tags, and metrics --\u001b[39;00m\n\u001b[1;32m   (...)\u001b[0m\n\u001b[0;32m    101\u001b[0m \n\u001b[0;32m    102\u001b[0m \u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[0;32m    103\u001b[0m _validate_run_id(run_id)\n\u001b[1;32m--> 104\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mstore\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mget_run\u001b[49m\u001b[43m(\u001b[49m\u001b[43mrun_id\u001b[49m\u001b[43m)\u001b[49m\n",
+      "File \u001b[1;32me:\\VM\\miniconda3\\envs\\mlops\\lib\\site-packages\\mlflow\\store\\tracking\\sqlalchemy_store.py:640\u001b[0m, in \u001b[0;36mSqlAlchemyStore.get_run\u001b[1;34m(self, run_id)\u001b[0m\n\u001b[0;32m    634\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mget_run\u001b[39m(\u001b[38;5;28mself\u001b[39m, run_id):\n\u001b[0;32m    635\u001b[0m     \u001b[38;5;28;01mwith\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mManagedSessionMaker() \u001b[38;5;28;01mas\u001b[39;00m session:\n\u001b[0;32m    636\u001b[0m         \u001b[38;5;66;03m# Load the run with the specified id and eagerly load its summary metrics, params, and\u001b[39;00m\n\u001b[0;32m    637\u001b[0m         \u001b[38;5;66;03m# tags. These attributes are referenced during the invocation of\u001b[39;00m\n\u001b[0;32m    638\u001b[0m         \u001b[38;5;66;03m# ``run.to_mlflow_entity()``, so eager loading helps avoid additional database queries\u001b[39;00m\n\u001b[0;32m    639\u001b[0m         \u001b[38;5;66;03m# that are otherwise executed at attribute access time under a lazy loading model.\u001b[39;00m\n\u001b[1;32m--> 640\u001b[0m         run \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_get_run\u001b[49m\u001b[43m(\u001b[49m\u001b[43mrun_uuid\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mrun_id\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43msession\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43msession\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43meager\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mTrue\u001b[39;49;00m\u001b[43m)\u001b[49m\n\u001b[0;32m    641\u001b[0m         mlflow_run \u001b[38;5;241m=\u001b[39m run\u001b[38;5;241m.\u001b[39mto_mlflow_entity()\n\u001b[0;32m    642\u001b[0m         \u001b[38;5;66;03m# Get the run inputs and add to the run\u001b[39;00m\n",
+      "File \u001b[1;32me:\\VM\\miniconda3\\envs\\mlops\\lib\\site-packages\\mlflow\\store\\tracking\\sqlalchemy_store.py:524\u001b[0m, in \u001b[0;36mSqlAlchemyStore._get_run\u001b[1;34m(self, session, run_uuid, eager)\u001b[0m\n\u001b[0;32m    519\u001b[0m runs \u001b[38;5;241m=\u001b[39m (\n\u001b[0;32m    520\u001b[0m     session\u001b[38;5;241m.\u001b[39mquery(SqlRun)\u001b[38;5;241m.\u001b[39moptions(\u001b[38;5;241m*\u001b[39mquery_options)\u001b[38;5;241m.\u001b[39mfilter(SqlRun\u001b[38;5;241m.\u001b[39mrun_uuid \u001b[38;5;241m==\u001b[39m run_uuid)\u001b[38;5;241m.\u001b[39mall()\n\u001b[0;32m    521\u001b[0m )\n\u001b[0;32m    523\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mlen\u001b[39m(runs) \u001b[38;5;241m==\u001b[39m \u001b[38;5;241m0\u001b[39m:\n\u001b[1;32m--> 524\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m MlflowException(\u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mRun with id=\u001b[39m\u001b[38;5;132;01m{\u001b[39;00mrun_uuid\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m not found\u001b[39m\u001b[38;5;124m\"\u001b[39m, RESOURCE_DOES_NOT_EXIST)\n\u001b[0;32m    525\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mlen\u001b[39m(runs) \u001b[38;5;241m>\u001b[39m \u001b[38;5;241m1\u001b[39m:\n\u001b[0;32m    526\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m MlflowException(\n\u001b[0;32m    527\u001b[0m         \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mExpected only 1 run with id=\u001b[39m\u001b[38;5;132;01m{\u001b[39;00mrun_uuid\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m. Found \u001b[39m\u001b[38;5;132;01m{\u001b[39;00m\u001b[38;5;28mlen\u001b[39m(runs)\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m.\u001b[39m\u001b[38;5;124m\"\u001b[39m,\n\u001b[0;32m    528\u001b[0m         INVALID_STATE,\n\u001b[0;32m    529\u001b[0m     )\n",
+      "\u001b[1;31mMlflowException\u001b[0m: Run with id=a930976208fa4fb48dbbf5cf4d7f600f not found"
+     ]
+    }
+   ],
+   "source": [
+    "import mlflow\n",
+    "from mlflow.tracking import MlflowClient\n",
+    "\n",
+    "# Set the tracking URI to the SQLite database\n",
+    "mlflow.set_tracking_uri(\"sqlite:///E:/Projects/Bioma-AI/time-series-forecasting/trend_prediction_app/trend-prediction-ml/trend-prediction-pipeline/mlruns.db\")\n",
+    "\n",
+    "# Initialize MLflow client\n",
+    "client = MlflowClient()\n",
+    "\n",
+    "# Get run information using the run ID\n",
+    "run_id = \"a930976208fa4fb48dbbf5cf4d7f600f\"\n",
+    "run = client.get_run(run_id)\n",
+    "print(f\"Run ID: {run.info.run_id}\")\n",
+    "print(f\"Run Status: {run.info.status}\")\n",
+    "print(f\"Artifact URI: {run.info.artifact_uri}\")\n",
+    "\n",
+    "# List artifacts in the run\n",
+    "artifacts = client.list_artifacts(run_id)\n",
+    "for artifact in artifacts:\n",
+    "    print(f\"Artifact Path: {artifact.path}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Run ID: 154301663fcc4245a7cc98eea0d123c3\n"
+     ]
+    }
+   ],
+   "source": [
+    "import mlflow\n",
+    "\n",
+    "# Set the tracking URI to your SQLite database\n",
+    "mlflow.set_tracking_uri(\"sqlite:///mlruns.db\")\n",
+    "\n",
+    "# Start a new run\n",
+    "with mlflow.start_run() as run:\n",
+    "    print(f\"Run ID: {run.info.run_id}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Tables in database: [('experiments',), ('alembic_version',), ('experiment_tags',), ('tags',), ('registered_models',), ('runs',), ('registered_model_tags',), ('model_version_tags',), ('model_versions',), ('latest_metrics',), ('metrics',), ('registered_model_aliases',), ('datasets',), ('inputs',), ('input_tags',), ('params',), ('trace_info',), ('trace_tags',), ('trace_request_metadata',)]\n",
+      "Contents of experiments:\n",
+      "(0, 'Default', 'file:///E:/Projects/Bioma-AI/time-series-forecasting/trend_prediction_app/trend-prediction-ml/trend-prediction-pipeline/mlruns/0', 'active', 1725160384768, 1725160384768)\n",
+      "Contents of alembic_version:\n",
+      "('4465047574b1',)\n",
+      "Contents of experiment_tags:\n",
+      "Contents of tags:\n",
+      "Contents of registered_models:\n",
+      "Contents of runs:\n",
+      "Contents of registered_model_tags:\n",
+      "Contents of model_version_tags:\n",
+      "Contents of model_versions:\n",
+      "Contents of latest_metrics:\n",
+      "Contents of metrics:\n",
+      "Contents of registered_model_aliases:\n",
+      "Contents of datasets:\n",
+      "Contents of inputs:\n",
+      "Contents of input_tags:\n",
+      "Contents of params:\n",
+      "Contents of trace_info:\n",
+      "Contents of trace_tags:\n",
+      "Contents of trace_request_metadata:\n"
+     ]
+    }
+   ],
+   "source": [
+    "import sqlite3\n",
+    "\n",
+    "db_path = r\"E:\\Projects\\Bioma-AI\\time-series-forecasting\\trend_prediction_app\\trend-prediction-ml\\trend-prediction-pipeline\\mlruns.db\"\n",
+    "try:\n",
+    "    with sqlite3.connect(db_path) as conn:\n",
+    "        cursor = conn.cursor()\n",
+    "        cursor.execute(\"SELECT name FROM sqlite_master WHERE type='table';\")\n",
+    "        tables = cursor.fetchall()\n",
+    "        print(\"Tables in database:\", tables)\n",
+    "        \n",
+    "        for table in tables:\n",
+    "            print(f\"Contents of {table[0]}:\")\n",
+    "            cursor.execute(f\"SELECT * FROM {table[0]} LIMIT 10;\")\n",
+    "            rows = cursor.fetchall()\n",
+    "            for row in rows:\n",
+    "                print(row)\n",
+    "except sqlite3.Error as e:\n",
+    "    print(f\"SQLite error: {e}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "env: MLFLOW_TRACKING_URI=sqlite:///mlruns.db\n"
+     ]
+    }
+   ],
+   "source": [
+    "%env MLFLOW_TRACKING_URI=sqlite:///mlruns.db"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "mlops",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.19"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

experimentation_mlops/mlops/train.py

@@ -0,0 +1,166 @@
+## TODO
+## Make the cross validation model get saved as well and log the params and model as mlflow artifacts
+
+import pandas as pd
+from modules.neuralforecast.core import NeuralForecast
+from modules.neuralforecast.models import TSMixer, TSMixerx, NHITS, MLPMultivariate, NBEATSx
+from modules.neuralforecast.losses.pytorch import MSE, MAE, MAPE
+from sklearn.preprocessing import StandardScaler
+import numpy as np
+import os
+
+horizon = 30
+input_size = horizon*2
+models = [
+    TSMixer(h=horizon,
+                input_size=input_size,
+                n_series=1,
+                max_steps=1000,
+                val_check_steps=100,
+                early_stop_patience_steps=5,
+                scaler_type='identity',
+                loss=MAPE(),
+                valid_loss=MAPE(),
+                random_seed=12345678,
+                ),
+    TSMixerx(h=horizon,
+                input_size=input_size,
+                n_series=1,
+                max_steps=1000,
+                val_check_steps=100,
+                early_stop_patience_steps=5,
+                scaler_type='identity',
+                dropout=0.7,
+                loss=MAPE(),
+                valid_loss=MAPE(),
+                random_seed=12345678,
+                futr_exog_list=['Gas', 'DXY', 'BrFu', 'BrDa'],
+                ),
+    NBEATSx(h=horizon,
+                input_size=horizon,
+                max_steps=1000,
+                val_check_steps=100,
+                early_stop_patience_steps=5,
+                scaler_type='identity',
+                loss=MAPE(),
+                valid_loss=MAPE(),
+                random_seed=12345678,
+                futr_exog_list=['Gas', 'DXY', 'BrFu', 'BrDa']
+                ),
+]
+
+nf = NeuralForecast(
+    models=models,
+    freq='D')
+
+Y_hat_df = nf.cross_validation(df=df,
+                               val_size=val_size,
+                               test_size=test_size,
+                               n_windows=None
+                               )
+Y_hat_df = Y_hat_df.reset_index()
+
+# Start from here
+
+
+# Using MLflow
+# Log your neuralforecast experiments to MLflow
+
+# Installing dependencies
+# To install Neuralforecast refer to https://nixtlaverse.nixtla.io/neuralforecast/examples/installation.html.
+
+# To install mlflow: pip install mlflow
+
+# Imports
+
+import logging
+import os
+import warnings
+
+import matplotlib.pyplot as plt
+import mlflow
+import mlflow.data
+import numpy as np
+import pandas as pd
+from mlflow.client import MlflowClient
+from mlflow.data.pandas_dataset import PandasDataset
+from utilsforecast.plotting import plot_series
+
+from neuralforecast.core import NeuralForecast
+from neuralforecast.models import NBEATSx
+from neuralforecast.utils import AirPassengersDF
+from neuralforecast.losses.pytorch import MAE
+     
+
+os.environ['NIXTLA_ID_AS_COL'] = '1'
+logging.getLogger("mlflow").setLevel(logging.ERROR)
+logging.getLogger("pytorch_lightning").setLevel(logging.ERROR)
+warnings.filterwarnings("ignore")
+     
+# Splitting the data
+
+# Split data and declare panel dataset
+Y_df = AirPassengersDF
+Y_train_df = Y_df[Y_df.ds<='1959-12-31'] # 132 train
+Y_test_df = Y_df[Y_df.ds>'1959-12-31'] # 12 test
+Y_df.tail()
+     
+# unique_id	ds	y
+# 139	1.0	1960-08-31	606.0
+# 140	1.0	1960-09-30	508.0
+# 141	1.0	1960-10-31	461.0
+# 142	1.0	1960-11-30	390.0
+# 143	1.0	1960-12-31	432.0
+# MLflow UI
+# Run the following command from the terminal to start the UI: mlflow ui. You can then go to the printed URL to visualize the experiments.
+
+# Model training
+
+mlflow.pytorch.autolog(checkpoint=False)
+
+with mlflow.start_run() as run:
+    # Log the dataset to the MLflow Run. Specify the "training" context to indicate that the
+    # dataset is used for model training
+    dataset: PandasDataset = mlflow.data.from_pandas(Y_df, source="AirPassengersDF")
+    mlflow.log_input(dataset, context="training")
+
+    # Define and log parameters
+    horizon = len(Y_test_df)
+    model_params = dict(
+        input_size=1 * horizon,
+        h=horizon,
+        max_steps=300,  
+        loss=MAE(),
+        valid_loss=MAE(),  
+        activation='ReLU',
+        scaler_type='robust',
+        random_seed=42,
+        enable_progress_bar=False,
+    )
+    mlflow.log_params(model_params)
+
+    # Fit NBEATSx model
+    models = [NBEATSx(**model_params)]
+    nf = NeuralForecast(models=models, freq='M')           
+    train = nf.fit(df=Y_train_df, val_size=horizon)
+    
+    # Save conda environment used to run the model
+    mlflow.pytorch.get_default_conda_env()
+    
+    # Save pip requirements
+    mlflow.pytorch.get_default_pip_requirements()
+
+mlflow.pytorch.autolog(disable=True)
+
+# Save the neural forecast model
+nf.save(path='./checkpoints/test_run_1/',
+        model_index=None, 
+        overwrite=True,
+        save_dataset=True)
+     
+#Seed set to 42
+#Forecasting the future
+
+Y_hat_df = nf.predict(futr_df=Y_test_df)
+plot_series(Y_train_df, Y_hat_df, palette='tab20b')
+     

experimentation_mlops/mlops/transform.py

@@ -0,0 +1,85 @@
+"""
+Split data into train-test-val based on proportions
+"""
+
+from modules.transformations import logReturn, scaleStandard, createLag
+
+import pandas as pd
+import numpy as np
+import click
+
+import mlflow
+
+@click.command(
+    help="Transforms the data to based on customizations"
+    "in an mlflow artifact called 'ratings-parquet-dir'"
+)
+@click.option("--data-parquet")
+@click.option(
+    "--test-size", default=0.1, help="Proportion of data for test set"
+)
+@click.option(
+    "--val-size", default=0.1, help="Proportion of data for test set"
+)
+
+def split(data_parquet, test_size=0.1, val_size=0.1, log_return = [], standard_scale = []):
+    """
+    Splits data into train-test-validation sets
+
+    Input:
+        :param pd.DataFrame data: Dataset for splitting
+        :param float test_size: Proportion of data for test set
+        :param float val_size: Proportiion of data for validation set
+
+    Output:
+        Split data into train-test-val
+    """
+    with mlflow.start_run():
+        y_log_ret = False
+        y_std_scale = False
+
+        data = pd.read_parquet(data_parquet)
+
+        # Split data into train-test-val without any randomization
+
+        train_size = int(len(data) * (1 - test_size - val_size))
+        test_size = int(len(data) * test_size)
+        val_size = int(len(data) * val_size)
+
+        train = data[:train_size].sort_values(by='ds', inplace=True)
+        test = data[train_size:train_size+test_size].sort_values(by='ds', inplace=True)
+        val = data[train_size+test_size:train_size+test_size+val_size].sort_values(by='ds', inplace=True)
+
+        # Transform select columns
+
+        if len(log_return) != 0:
+            
+            for col1 in log_return:
+                try:
+                    #print(data[col1])
+                    data[col1] = logReturn(data, col1)
+                except Exception as e:
+                    print(e)
+                    pass
+                
+            if 'y' in log_return:
+                y_log_ret = True
+
+        if len(standard_scale) != 0:
+            
+            for col2 in standard_scale:
+                try:
+                    data[col2], _ = scaleStandard(data[[col2]])
+                except Exception as e:
+                    print(e)
+                    pass
+                
+            if 'y' in standard_scale:
+                data['y'], yScaler = scaleStandard(data[['y']])
+                y_std_scale = True
+
+        return data
+
+
+if __name__ == "__main__":
+    split()

modules/data_preparation.py

@@ -0,0 +1,86 @@
+import re
+import plotly.express as px
+import datetime
+import plotly.graph_objects as go
+import numpy as np
+import pandas as pd
+import datetime
+
+def clean_text(text):
+    new_text = text
+    for rgx_match in ['[A-Z ]+:']:
+        new_text = re.sub(rgx_match, '', new_text)
+    return new_text
+
+def prepare_df(df, categories, date_filter):
+    try:
+        df.drop(columns=['Unnamed: 0'], inplace=True)
+    except:
+        pass
+
+    #df['topic_verification'][(df.headline.str.contains('crude', case=False)) | df.body.str.contains('crude', case=False)] = 'Crude Oil'
+
+    try:
+        news_data = df[df['topic_verification'].isin(categories)]
+
+        actual_day = datetime.date.today() - datetime.timedelta(days=1)
+        pattern_del = actual_day.strftime('%b').upper()
+
+        filter = news_data['headline'].str.contains(pattern_del)
+        news_data = news_data[~filter]
+
+        # shift column 'C' to first position 
+        first_column = news_data.pop('headline') 
+        
+        # insert column using insert(position,column_name,first_column) function 
+        news_data.insert(0, 'headline', first_column) 
+
+        news_data['updatedDate'] = pd.to_datetime(news_data['updatedDate'], format='%Y-%m-%d %H:%M:%S%z')
+
+        dates = []
+
+        dates.append(datetime.datetime.strftime(date_filter[0], '%Y-%m-%d %H:%M:%S%z'))
+        dates.append(datetime.datetime.strftime(date_filter[1], '%Y-%m-%d %H:%M:%S%z'))
+
+        news_data = news_data[(news_data['updatedDate'] >= dates[0]) & (news_data['updatedDate'] <= dates[1])]
+
+    except Exception as E:
+        print(E)
+
+    return news_data
+
+def plot_3dgraph(news_data):
+    fig = px.scatter_3d(news_data, 
+                        x='neutral_score', 
+                        y='negative_score', 
+                        z='positive_score',
+                        color='positive_score', 
+                        hover_name ='headline', 
+                        color_continuous_scale='RdBu', 
+                        size_max=40,
+                        size='negative_score',
+                        #text='headline',
+                        hover_data='topic_verification')
+
+    fig.update_layout(
+        height=600,
+        title=dict(text=f"News Semantics towards Crude Oil Price <br><sup>Hover cursor on a datapoint to show news title</sup>", 
+                   font=dict(size=35), 
+                   automargin=False)
+    )
+
+    fig.update_traces(textfont_size=8)
+
+    trace=dict(type='scatter3d',
+           x=news_data.iloc[[-1]]['neutral_score'],
+           y=news_data.iloc[[-1]]['negative_score'],
+           z=news_data.iloc[[-1]]['positive_score'],
+           mode='markers',
+           name= 'MEAN OF SELECTED NEWS',
+           marker=dict(color=[f'rgb({0}, {250}, {200})' for _ in range(25)],
+                       size=10)
+          )
+
+    fig.add_trace(trace)
+
+    return fig

modules/semantic.py

@@ -0,0 +1,198 @@
+import pandas as pd
+import os
+import nltk
+from nltk.corpus import stopwords
+import plotly.express as px
+from collections import Counter
+import re
+import matplotlib.pyplot as plt
+from wordcloud import WordCloud
+
+place_mapping = {
+    'united states': 'United States',
+    'u.s.': 'United States',
+    'US': 'United States',
+    'america': 'United States',
+    'north america': 'North America',
+    'usa': 'United States',
+    'south america': 'South America',
+    'american': 'United States',
+    'europe': 'Europe',
+    'eu': 'Europe',
+    'china': 'China',
+    'chinese': 'China',
+    'russia': 'Russia',
+    'arab': 'Arab Countries',
+    'middle east': 'Middle East',
+    'asia': 'Asia',
+    'asian': 'Asia',
+    'spain': 'Spain',
+    'germany': 'Germany',
+    'france': 'France',
+    'uk': 'United Kingdom',
+    'britain': 'United Kingdom',
+    'canada': 'Canada',
+    'mexico': 'Mexico',
+    'brazil': 'Brazil',
+    'venezuela': 'Venezuela',
+    'angola': 'Angola',
+    'nigeria': 'Nigeria',
+    'libya': 'Libya',
+    'iraq': 'Iraq',
+    'iran': 'Iran',
+    'kuwait': 'Kuwait',
+    'qatar': 'Qatar',
+    'saudi arabia': 'Saudi Arabia',
+    'gcc': 'Gulf Cooperation Council',
+    'asia-pacific': 'Asia',
+    'southeast asia': 'Asia',
+    'latin america': 'Latin America',
+    'caribbean': 'Caribbean',
+}
+
+region_mapping = {
+    'North America': ['United States', 'Canada', 'Mexico'],
+    'South America': ['Brazil', 'Venezuela'],
+    'Europe': ['United Kingdom', 'Germany', 'France', 'Spain', 'Russia'],
+    'Asia': ['China', 'India', 'Japan', 'South Korea'],
+    'Middle East': ['Saudi Arabia', 'Iran', 'Iraq', 'Qatar', 'Kuwait'],
+    'Africa': ['Nigeria', 'Libya', 'Angola'],
+    # Add more regions as necessary
+}
+
+
+nomenclature_mapping = {
+    'petroleum': 'Petroleum',
+    'energy': 'Energy',
+    'fuel oil': 'Fuel Oil',
+    'shale': 'Shale',
+    'offshore': 'Offshore',
+    'upstream': 'Upstream',
+    'hsfo': 'HSFO',
+    'downstream': 'Downstream',
+    'crude oil': 'Crude Oil',
+    'crude' : 'Crude Oil',
+    'refinery': 'Refinery',
+    'oil field': 'Oil Field',
+    'drilling': 'Drilling',
+    'gas': 'Gas',
+    'liquefied natural gas': 'LNG',
+    'natural gas': 'NG',
+    'oil': 'Crude Oil',
+}
+
+company_mapping = {
+    'exxonmobil': 'ExxonMobil',
+    'exxon': 'ExxonMobil',
+    'chevron': 'Chevron',
+    'bp': 'BP',
+    'british petroleum': 'BP',
+    'shell': 'Shell',
+    'total energies': 'TotalEnergies',
+    'conoco': 'ConocoPhillips',
+    'halliburton': 'Halliburton',
+    'slb': 'SLB',
+    'schlumberger': 'SLB',
+    'devon': 'Devon Energy',
+    'occidental': 'Occidental Petroleum',
+    'marathon': 'Marathon Oil',
+    'valero': 'Valero Energy',
+    'aramco': 'Aramco',
+}
+
+nltk.download('stopwords')
+
+stop_words = set(stopwords.words('english'))
+
+
+# Function to clean, tokenize, and remove stopwords
+def tokenize(text):
+
+    text = re.sub(r'[^\w\s]', '', text.lower())
+    words = text.split()
+
+    mapped_words = []
+    for word in words:
+        mapped_word = place_mapping.get(word, 
+                        nomenclature_mapping.get(word, 
+                        company_mapping.get(word, word)))
+        mapped_words.append(mapped_word)
+
+    filtered_words = [word for word in mapped_words if word not in stop_words]
+    return filtered_words
+
+
+# Function to apply filtering and plotting based on search input
+def generateChartBar(data, search_word, body=False):
+
+    filtered_df = data[data['headline'].str.contains(search_word, case=False) | data['body'].str.contains(search_word, case=False)]
+    
+    all_words = []
+    filtered_df['headline'].apply(lambda x: all_words.extend(tokenize(x)))
+
+    if body:
+        filtered_df['body'].apply(lambda x: all_words.extend(tokenize(x)))
+    
+
+    word_counts = Counter(all_words)
+    top_10_words = word_counts.most_common(20)
+    top_10_df = pd.DataFrame(top_10_words, columns=['word', 'frequency'])
+
+    fig = px.bar(top_10_df, x='word', y='frequency', title=f'Top 20 Most Common Words (Excluding Stopwords) for "{search_word}"',
+                 labels={'word': 'Word', 'frequency': 'Frequency'}, 
+                 text='frequency')
+    
+    return fig
+
+# Function to filter based on the whole word/phrase and region
+def filterPlace(data, search_place):
+    # Check if the search_place is a region
+    if search_place in region_mapping:
+        # Get all countries in the region
+        countries_in_region = region_mapping[search_place]
+        # Map countries to their place_mapping synonyms
+        synonyms_pattern = '|'.join(
+            r'\b{}\b'.format(re.escape(key))
+            for country in countries_in_region
+            for key in place_mapping
+            if place_mapping[key] == country
+        )
+    else:
+        # If a country is selected, get its standard place and synonyms
+        standard_place = place_mapping.get(search_place.lower(), search_place)
+        synonyms_pattern = '|'.join(
+            r'\b{}\b'.format(re.escape(key)) 
+            for key in place_mapping 
+            if place_mapping[key] == standard_place
+        )
+    
+    # Filter the DataFrame for headlines or body containing the whole word/phrase
+    filtered_df = data[
+        data['headline'].str.contains(synonyms_pattern, case=False, na=False) | 
+        data['body'].str.contains(synonyms_pattern, case=False, na=False)
+    ]
+    
+    if filtered_df.empty:
+        print(f'No data found for {search_place}. Please try a different location or region.')
+        return None
+    
+    return filtered_df
+
+# Function to filter DataFrame and generate a word cloud
+def generateWordCloud(data):
+
+    # standard_place = place_mapping.get(search_place.lower(), search_place)
+    # synonyms_pattern = '|'.join(re.escape(key) for key in place_mapping if place_mapping[key] == standard_place)
+    
+    # filtered_df = data[data['headline'].str.contains(synonyms_pattern, case=False, na=False) | 
+    #                  data['body'].str.contains(synonyms_pattern, case=False, na=False)]
+    
+    # if filtered_df.empty:
+    #     print(f'No data found for {search_place}. Please try a different location.')
+    #     return
+    
+    text = ' '.join(data['headline'].tolist() + data['body'].tolist())
+    wordcloud = WordCloud(width=800, height=400, background_color='white').generate(text)
+    
+ 
+    return wordcloud

page_1.py

@@ -0,0 +1,85 @@
+import streamlit as st
+import pandas as pd
+from modules.data_preparation import prepare_df, plot_3dgraph
+import numpy as np
+import matplotlib.pyplot as plt
+from datetime import datetime
+from modules.semantic import generateChartBar, generateWordCloud, filterPlace
+
+st.title('Semantic Analysis for Price Trend Prediction - Crude Oil Futures')
+
+st.header('Filter news based on categories and country/region')
+
+# st.header(f'Data based on News Data')
+# st.subheader(f'{datetime.now()}')
+
+date_filter = st.slider(
+    "Date Filter",
+    value=(datetime(2024, 8, 4), datetime(2024,8,9)),
+    format="MM/DD/YY",
+)
+
+col1, col2 = st.columns(2)
+
+with col1:
+    news_categories = st.multiselect("Select desired news categories", 
+                                 ["Macroeconomic & Geopolitics", "Crude Oil", "Light Ends", "Middle Distillates", "Heavy Distillates", "Other"],
+                                 ["Macroeconomic & Geopolitics", "Crude Oil"])
+
+
+with col2:
+    news_location = st.selectbox("Select desired mentioned location", 
+                                 ["North America","United States", "Russia", "Asia", "Europe"])
+
+st.subheader('Tabular Data')
+
+
+latest_news = prepare_df(pd.read_excel('evaluation.xlsx'), news_categories, date_filter)
+df_news = pd.concat([latest_news], ignore_index=True).drop_duplicates(['headline'])
+df_news = filterPlace(df_news, news_location)
+
+df_mean = pd.DataFrame({
+    'headline' : ['MEAN OF SELECTED NEWS'],
+    'negative_score' : [df_news['negative_score'].mean()],
+    'neutral_score' : [df_news['neutral_score'].mean()],
+    'positive_score' : [df_news['positive_score'].mean()],
+    'topic_verification' : ['']
+})
+
+df_news_final = pd.concat([df_news, df_mean])
+
+df_news_final.index = np.arange(1, len(df_news_final) + 1)
+
+st.dataframe(df_news_final.iloc[:, : 9])
+
+try:
+    st.plotly_chart(plot_3dgraph(df_news_final), use_container_width=True)
+except:
+    st.subheader('Select news categories to plot 3D graph')
+
+st.markdown('---')
+
+viz1, viz2 = st.columns(2)
+
+st.subheader('Top Word Frequency - Bar Chart')
+
+bar_chart = generateChartBar(data=df_news,search_word='n', body=True)
+st.plotly_chart(bar_chart)
+
+
+st.markdown('---')
+
+st.subheader('Top Word Frequency - Word Cloud')
+
+wordcloud = generateWordCloud(data=df_news)
+
+# Display the generated image:
+fig, ax = plt.subplots()
+ax.imshow(wordcloud, interpolation='bilinear')
+ax.axis("off")
+st.pyplot(fig)
+
+st.markdown('---')
+
+st.subheader('Other possible use cases:')
+st.markdown('- Sentiments towards a company, country, or individual')

page_2.py

@@ -0,0 +1,63 @@
+import streamlit as st
+import pandas as pd
+import os
+from PIL import Image
+
+st.title('Price Forecasting - Crude Oil Futures')
+st.subheader('This page is not interactive - only for prototype purposes*')
+st.text('*Due to not having access to GPU for cloud computation yet.')
+
+st.header('Univariate Forecasting with Exogenous Predictors')
+
+col1, col2, col3 = st.columns(3)
+
+uni_df = pd.read_csv(os.path.join('price_forecasting_ml', 
+                                   'artifacts',
+                                   'crude_oil_8998a364-2ecc-483d-8079-f04d455b4522',
+                                   'train_data.csv')).drop(columns=['Unnamed: 0'])
+
+with col1:
+    horizon_uni = st.text_input('Univariate Forecasting Horizon')
+with col2:
+    target_uni = st.multiselect('Univariate Target Variable', uni_df.columns
+                             ,default='y')
+with col3:
+    agg_uni = st.selectbox('Univariate Data Aggregation', 
+                           ['Daily', 'Weekly', 'Monthly', 'Yearly'])
+
+
+st.dataframe(uni_df)
+
+img1 = Image.open(os.path.join('price_forecasting_ml',
+                              'artifacts',
+                              'crude_oil_8998a364-2ecc-483d-8079-f04d455b4522',
+                              'forecast_plot.jpg'))
+st.image(img1, caption="Crude Oil Futures Price Forecasting - Univariate with Exogenous Features (Horizon = 5)")
+
+st.markdown("---")
+
+st.header('Multivariate Forecasting')
+
+col4, col5, col6 = st.columns(3)
+
+multi_df = pd.read_csv(os.path.join('price_forecasting_ml', 
+                                   'artifacts',
+                                   'crude_oil_df1ce299-117d-43c7-bcd5-7ecaeac0bc89',
+                                   'train_data.csv')).drop(columns=['Unnamed: 0'])
+
+with col4:
+    horizon_multi = st.text_input('Multivariate Forecasting Horizon')
+with col5:
+    target_multi = st.multiselect('Multivariate Target Variable', multi_df.columns
+                             ,default='y')
+with col6:
+    agg_multi = st.selectbox('Multivariate Data Aggregation', 
+                           ['Daily', 'Weekly', 'Monthly', 'Yearly'])
+
+st.dataframe(multi_df)
+
+img2 = Image.open(os.path.join('price_forecasting_ml',
+                              'artifacts',
+                              'crude_oil_df1ce299-117d-43c7-bcd5-7ecaeac0bc89',
+                              'forecast_plot.jpg'))
+st.image(img2, caption="Crude Oil Futures Price Forecasting - Univariate with Exogenous Features (Horizon = 5)")

page_3.py

@@ -0,0 +1,79 @@
+import streamlit as st
+from PIL import Image
+import os
+
+st.title('Machine Learning Operations Pipeline')
+
+st.markdown("""
+# Machine Learning Operations (MLOps) Pipeline Documentation
+
+This is the documentation covering  each of the steps included in Bioma AI's time-series-forecasting MLOps Pipeline.
+
+## Sequential MLOps Steps
+The information flow of the pipeline will closely resemble that of a regression machine learning task. The model development will consist of sequential steps:
+1. Ingestion,
+2. Transformation,
+3. Training,
+4. Evaluation, and
+5. Registration.
+
+""")
+
+img = Image.open(os.path.join('experimentation_mlops',
+                              'mlops',
+                              'pics',
+                              'pipeline.png'))
+st.image(img, caption="MLOps Pipeline for Bioma AI")
+            
+st.markdown("""
+
+## 1. Ingestion
+
+Our pipeline involves extracting raw datasets from the internet (S3 Buckets and other cloud services), the assumed dataset is of one of the following file types: csv, json, parquet or xlsx. The extracted data is saved as an artifact which can help in documentation purposes.
+
+In the case of time series forecasting, the data ingestion process is tasked on receiving data from a specific format and converting it to a Pandas Dataframe for further processing. The data will be downloaded from the web by issuing a request, the data will then be converted into parquet before being written as a Pandas dataframe. The parquet file will be saved as an artifact for the purpose of documentation.
+
+## 2. Transformation
+
+According to the timeframe of the time-series data, the data will be split into a train-test-validation set. The user will be able to customize each of the set's proportions.
+
+Various statistical methods is considered and performed into a selection of columns, the columns and the methods are both customizable. A few methods that are considered are:
+1. Logarithmic
+2. Natural Logarithmic
+3. Standardization
+4. Identity
+5. Logarithmic Difference
+
+## 3. Training
+
+The training process can be broken down into two types according to the amount of variates being predicted: univariate or multivariate.
+
+Predictors are either an:
+
+1. Endogenous feature (Changes in the target's value has an effect on the predictor's value or the other way around) or 
+2. Exogenous feature (changes in the predictor's value has an effect on the target's value, but not the other way around)
+    <ol type="a">
+    <li>Static Exogenous</li>
+    Static variables such as one-hot encoding for a categorical class identifier.
+    <li>Historical Exogenous</li>
+    Exogenous features that their historical data is only known of.
+    <li>Future Exogenous</li>
+    Exogenous features that their data is known of when making the prediction on that time in the future.
+    </ol>
+
+Endogenous features will be predicted in conjunction with the target's feature. Exogenous features will not be predicted, rather only be used to predict the target variable.
+
+In short: multivariate predictions will use predictors as endogenous features, while multivariable predictions use predictors as exogenous features because of its univariate nature.
+
+## 4. Evaluation
+
+The evaluation step is constructed for the trained models to perform prediction on out-of-training data. Ideally, this step will produce outputs such as visualizations and error metrics for arbitrary datasets. 
+
+## 5. Registration
+            
+Registration includes saving the model with the highest accuracy, making it easy to retrieve for inference later on.
+            
+References:
+- [1] [mlflow/recipes-regression-template](https://github.com/mlflow/recipes-regression-template/tree/main?tab=readme-ov-file#installation)
+- [2] [MLflow deployment using Docker, EC2, S3, and RDS](https://aws.plainenglish.io/set-up-mlflow-on-aws-ec2-using-docker-s3-and-rds-90d96798e555)
+            """)