satoshi-2000/llms-suitable huggingface dataset:
Question: Pandas Webscraping Errors

I'm currently trying to webscrape websites for tables using pandas and I get this error for one of the links.
Here's a snippet of what causes the crash:
import pandas as pd
website_df = pd.read_html("https://ballotpedia.org/Roger_Wicker")
print(website_df)

Below is the error I get, does anyone know how to fix this?
Traceback (most recent call last):
  File "C:\Users\miniconda3\lib\site-packages\pandas\io\parsers\python_parser.py", line 700, in _next_line
    line = self._check_comments([self.data[self.pos]])[0]
IndexError: list index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\miniconda3\lib\site-packages\pandas\io\parsers\python_parser.py", line 385, in _infer_columns
    line = self._next_line()
  File "C:\Users\miniconda3\lib\site-packages\pandas\io\parsers\python_parser.py", line 713, in _next_line
    raise StopIteration
StopIteration

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "c:\Users\legislators-current.py", line 15, in <module>
    website_df = pd.read_html("https://ballotpedia.org/Roger_Wicker")
  File "C:\Users\miniconda3\lib\site-packages\pandas\util\_decorators.py", line 331, in wrapper
    return func(*args, **kwargs)
  File "C:\Users\miniconda3\lib\site-packages\pandas\io\html.py", line 1205, in read_html
    return _parse(
  File "C:\Users\miniconda3\lib\site-packages\pandas\io\html.py", line 1011, in _parse
    df = _data_to_frame(data=table, **kwargs)
  File "C:\Users\miniconda3\lib\site-packages\pandas\io\html.py", line 890, in _data_to_frame
    with TextParser(body, header=header, **kwargs) as tp:
  File "C:\Users\miniconda3\lib\site-packages\pandas\io\parsers\readers.py", line 1876, in TextParser
    return TextFileReader(*args, **kwds)
  File "C:\Users\miniconda3\lib\site-packages\pandas\io\parsers\readers.py", line 1442, in __init__
    self._engine = self._make_engine(f, self.engine)
  File "C:\Users\miniconda3\lib\site-packages\pandas\io\parsers\readers.py", line 1753, in _make_engine
    return mapping[engine](f, **self.options)
  File "C:\Users\miniconda3\lib\site-packages\pandas\io\parsers\python_parser.py", line 122, in __init__
    ) = self._infer_columns()
  File "C:\Users\miniconda3\lib\site-packages\pandas\io\parsers\python_parser.py", line 395, in _infer_columns
    raise ValueError(
ValueError: Passed header=[1,2], len of 2, but only 2 lines in file



Answer:
Set header=0. You're going to get a lot of dataframes, but you can parse them to get what you need.
website_df = pd.read_html("https://ballotpedia.org/Roger_Wicker", header=0)
Question: Ansible - starting a task after cpu load is below 2.0

I am trying to create a playbook where I want to perform a simple debug task after cpu load is below 2.0.
I have this so far in cpu-load.yml:
---
- name: Check CPU load and wait
  hosts: localhost
  gather_facts: yes
  
  tasks:
    - name: Check cpu load
      shell: uptime | awk -F 'load average:' '{print $2}' | awk -F ', ' '{print $1}'
      register: cpu_load
      
    - name: Wait until cpu load is bellow 2.0
      wait_for:
        timeout: 300
        delay: 10
        shell: Do something here
        msg: "cpu load is bellow 2.0"
      
    - name: Continue jobs
      debug:
        msg: "CPU load is bellow 2.0. Continue!!!"

Now I am not sure how to make the task wait for the cpu load to go bellow 2.0. Can you guys help?


Answer:
You need to put an until loop around your "check cpu load" task:
- hosts: localhost
  gather_facts: false
  tasks:
    - name: Check cpu load
      shell: uptime | awk -F 'load average:' '{print $2}' | awk -F ', ' '{print $1}'
      register: cpu_load
      until: cpu_load.stdout|float < 2.0
      retries: 300
      delay: 1

    - name: Some other task
      debug:
        msg: hello world

This will wait up to five minutes (300 retries with a 1-second delay) for the load average to drop below 2.0.

There are probably better ways to get the current 1-minute CPU load; reading from /proc/loadavg may be easiest:
- hosts: localhost
  gather_facts: false
  tasks:
    - name: Check cpu load
      command: cat /proc/loadavg
      register: cpu_load
      until: cpu_load.stdout.split()|first|float < 2.0
      retries: 300
      delay: 1

    - name: Some other task
      debug:
        msg: hello world
Question: How can I use Webpack to load a separate JS file in Shopware 6 and improve web performance?

How to load a separate JS file in Shopware 6 using webpack?
What?
I'm trying to load a separate javascript file next to the all.js file by using WebPack.
Why?
The all.js file can get really big and you're loading unnecessary javascript on a page. So by using code splitting (which should be possible since WebPack is implemented in Shopware 6) and dynamic imports you could stop loading unnecessary javascript.
What I've tried
I've added a webpack.config.js file inside the root of my theme plugin like so:
module.exports = {

    entry: {
        main: './src/main.js', 
        separateFile: './src/js/separate.js'
    },

    output: {
        filename: '[name].js'
    },

    optimization: {
        splitChunks: {
            chunks: 'all',
        },
    },
};

After running bin/build-storefront.sh there is no separate JS file added in the public folder.
I'm also trying to dynamically load this JS file in src/Resources/app/storefront/src/main.js but this results in a 404 since the separate file doesn't exist.


Answer:
This will not work since all pre-compiled assets of plugins are collected in the ThemeCompiler and concatenated into one single script. This is done  in PHP since node is not a requirement for production environments.
You could try to add separate scripts as additional custom assets, but you would still have to extend the template to add the corresponding script tags manually.
Question: Chunked xarray: load only 1 cell in memory efficiently

Context:
I have a datacube with 3 variables (3D arrays, dims:time,y,x). The datacube is too big to fit in memory so I chunk it with xarray/dask. I want to apply a function to every cell in x,y of every variable in my datacube.
Problem:
My method takes a long time to load only one cell (1 minute) and I have to do that 112200 times. I use a for loop with dataset.variable.isel(x=i, y=j).values to load a single 1D array from my variables. Is there a better way to do that ? Also, knowing my dataset is chunked, is there a way to do that in parallel for all the chunks at once ?
Code example:
# Setup
import xarray as xr
import numpy as np

# Create the dimensions
x = np.linspace(0, 99, 100)
y = np.linspace(0, 349, 350)
time = np.linspace(0, 299, 300)

# Create the dataset
xrds= xr.Dataset()

# Add the dimensions to the dataset
xrds['time'] = time
xrds['y'] = y
xrds['x'] = x



# Create the random data variables with chunking
chunksize = (10, 100, 100)  # Chunk size for the variables
data_var1 = np.random.rand(len(time), len(y), len(x))
data_var2 = np.random.rand(len(time), len(y), len(x))
data_var3 = np.random.rand(len(time), len(y), len(x))

xrds['data_var1'] = (('time', 'y', 'x'), data_var1, {'chunks': chunksize})
xrds['data_var2'] = (('time', 'y', 'x'), data_var2, {'chunks': chunksize})
xrds['data_var3'] = (('time', 'y', 'x'), data_var3, {'chunks': chunksize})

#### ---- My Attempt ---- ####

# Iterate through all the variables in my dataset
for var_name, var_data in xrds.data_vars.items():

    # if variable is 3D
    if var_data.shape == (xrds.dims['time'], xrds.dims['y'], xrds.dims['x']):

        # Iterate through every cell of the variable along the x and y axis only
        for i in range(xrds.dims['y']):
            for j in range(xrds.dims['x']):

                # Load a single 1D cell into memory (len(cell) = len(time))
                print(xrds.v.isel(y=i,x=j).values)



Answer:
I find that explicitly iterating over the xarray is faster than isel(), by about 10%.
Example:
    for var_name, var_data in xrds.data_vars.items():

        # if variable is 3D
        if var_data.shape == (xrds.dims['time'], xrds.dims['y'], xrds.dims['x']):

            # Iterate through every cell of the variable along the x and y axis only
            for i_array in xrds['data_var1'].transpose('x', 'y', 'time'):
                x_coordinate = i_array.x.item()
                for cell in i_array.transpose('y', 'time'):
                    y_coordinate = cell.y.item()
                    # Do something with cell

This takes 17.38s, versus 20.47s for the original.
PS: The line chunksize = (10, 100, 100) seems very suspicious to me. It seems like if you want to load an array corresponding to the entire time axis at once, the chunks should be set so that this doesn't require looking at multiple chunks. It seems like chunksize = (len(time), 100, 100) would be more efficient. However, I benchmarked this both ways and it doesn't make a difference for this data size. May make a difference on your larger problem.
Question: How to change colors when using scale_fill_discrete in R?

I have the data below:
time=c(200,218,237,237,237,237,237,246,246,246,257,257,257,272,272,272,294,294,294)
location=c("A","A","D","C","A","B","B","D","C","B","D","C","B","D","C","B","D","C","B")
value=c(0,774,0,0,2178,0,2178,0,1494,2644,1326,1504,4188,3558,1385,5013,12860,829,3483)
dataA=data.frame(time,location,value)

and I made a graph.
ggplot(data=dataA, aes(x=time, y=value))+
  geom_area(aes(group=location, fill=location), position="stack", linetype=1, size=0.5 ,colour="black") +
  scale_fill_discrete(breaks=c("A","B","C","D"), labels=c("Main_town","B","C","D"))+
  theme_classic(base_size=18, base_family="serif")+
  theme(legend.position="right",
        axis.line=element_line(linewidth=0.5, colour="black"))+
windows(width=5.5, height=5)

I changed one of the legend label from A to main_town using scale_fill_discrete(). Then color is automatically generated.

I want to change this color according to my preference. When I add a code, scale_fill_manual(values=c("darkblue","darkred","khaki4","darkgreen"))+ the below message shows up and the graph is before I changed legend label.
Scale for fill is already present.
Adding another scale for fill, which will replace the existing scale.


How can I change colors when using scale_fill_discrete()? I want to change colors to "darkblue","darkred","khaki4","darkgreen"
Could you please let me know how to do that? Or do you let me know simply how to change legend labels maintaining colors I want?
Always many thanks!!!


Answer:
I think you need scale_fill_discrete(type = c(...)).
library(ggplot2)

ggplot(data=dataA, aes(x=time, y=value))+
  geom_area(aes(group=location, fill=location), position="stack", linetype=1, size=0.5 ,colour="black") +
  scale_fill_discrete(breaks=c("A","B","C","D"), labels=c("Main_town","B","C","D"),
                      type=c("darkblue","darkred","khaki4","darkgreen"))+
  theme_classic(base_size=18, base_family="serif")+
  theme(legend.position="right",
        axis.line=element_line(linewidth=0.5, colour="black"))
Question: How can I use Python to increment 'tokenId' values in a .json file?

I have a .json file but I got the tokenId numbering wrong. I need to increase all values of "tokenId" by 1 number
[
    {
        "Background": "Red",
        "Body": "Tunn",
        "Hat": "Bambu",
        "Outfit": "Pirate",
        "Expression": "Sad",
        "Accessory": "Rifle",
        "tokenId": 0
    },
    {
        "Background": "Lilac",
        "Body": "Tunn",
        "Hat": "Bicorn",
        "Outfit": "Pirate",
        "Expression": "Angry",
        "Accessory": "Balloons",
        "tokenId": 1
    },

          ...
    
    {
        "Background": "Green",
        "Body": "Tunn",
        "Hat": "Bicorn",
        "Outfit": "Pirate",
        "Expression": "Sad",
        "Accessory": "Balloons",
        "tokenId": 3000
    },


is it possible to do this with python? i created this .json file with python.
I tried this code, but I get an error
import json

with open('traits.json') as f:
    data = json.load(f)

for item in data['tokenId']:
    item['tokenId'] = item['tokenId'].replace([int('+1')])

with open('new_data.json', 'w') as f:
    json.dump(data, f)

TypeError: list indices must be integers or slices, not str

Thank you!


Answer:
To increase the values of the "tokenId" field in your JSON file by 1, you can modify your code as follows:
import json

with open('traits.json') as f:
    data = json.load(f)

for item in data:
    item['tokenId'] += 1

with open('new_data.json', 'w') as f:
    json.dump(data, f)

In your original code, you were trying to access data['tokenId'] as if it was a list, but it is actually a dictionary. Instead, you need to iterate over the list data and update the "tokenId" field of each item. By using item['tokenId'] += 1, you increment the value of "tokenId" by 1.
Finally, the modified data is saved to a new JSON file named "new_data.json" using json.dump(data, f).
After running this code, the "new_data.json" file will contain the updated "tokenId" values with an increment of 1.
Question: How to extract data from xml in NodeJS?

I have the following xml:
<?xml version="1.0" encoding="utf-8"?>
    <wfs:FeatureCollection xmlns:wfs="http://www.opengis.net/wfs/2.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation=" http://www.opengis.net/wfs/2.0 http://www.wfs.nrw.de/aaa-suite/schema/ogc/wfs/2.0/wfs.xsd" timeStamp="2023-06-01T13:31:53.444+02:00" numberReturned="0" numberMatched="9359426"/>

How can I extract the value of numberMatched using an xml parser like fast-xml-parser in NodeJS?


Answer:
You need to set ignoreAttributes option to false
import { XMLParser } from "fast-xml-parser";

const XMLdata = `<?xml version="1.0" encoding="utf-8"?>
<wfs:FeatureCollection xmlns:wfs="http://www.opengis.net/wfs/2.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation=" http://www.opengis.net/wfs/2.0 http://www.wfs.nrw.de/aaa-suite/schema/ogc/wfs/2.0/wfs.xsd" timeStamp="2023-06-01T13:31:53.444+02:00" numberReturned="0" numberMatched="9359426"/>`;

const parser = new XMLParser({
  ignoreAttributes: false,
  attributeNamePrefix: ""
});
let jObj = parser.parse(XMLdata);

console.log(jObj["wfs:FeatureCollection"].numberMatched);
Question: How to make a forever loop in JS not freeze

I'm working on a block based programming language based of of Google's Blockly. I need to make a block that loops the contents forever, for making games.
I tried a while (true) loop but it froze. Is there any way to make a forever loop that won't freeze and will let other scripts run?
Thanks!


Answer:
check setTimeout() : https://developer.mozilla.org/en-US/docs/Web/API/setTimeout
Something like that to loop indefinitely without blocking the main thread (you should probably design a way to break the loop at some point) :
function doSomeStuff() {
  // do some stuff…
  setTimeout(() => {
    doSomeStuff();
  }, 1000);
}
Question: What is the correct syntax to classify ages into groups using IF statements in Google Sheets?

[Why am i getting "formula parse error" when I try to classify the ages (column H) into groups using the following formula? And is there a better way? Thanks for your assistance:
=IF (H19<20, “0-19”, IF ((H19>=20 AND H19<40), “20-39”, IF ((H19>=40 AND H19<60), “40-59”, IF ((H19>=60 AND H19<70), “60-69”, IF (H19>=70, ">= 70", “WRONG”)))))
I was expecting to output the Age column into strings based on my category definitions.


Answer:
The portions that you have formatted as (H19>=20 AND H19<40) should be changed to AND(H19>=20, H19<40). Your final formula should then be:
=IF(H19<20, “0-19”,
  IF(AND(H19>=20, H19<40), “20-39”,
  IF(AND(H19>=40, H19<60), “40-59”,
  IF(AND(H19>=60, H19<70), “60-69”,
  IF(H19>=70, ">= 70", “WRONG”)))))

Alternatively:
=IFS(OR(NOT(ISNUMBER(H19)),H19<0), "WRONG",
  H19<20, "0-19",
  AND(H19>=20, H19<40), "20-39",
  AND(H19>=40, H19<60), "40-59",
  AND(H19>=60, H19<70), "60-69",
  H19>=70, ">= 70")
Question: Masking a pandas column based on another column with slightly different values

So what I have is two Pandas dataframes in Python with a large number of xyz-coordinates. One of them will be used to mask/remove some coordinates in the other one, but the problem is that the coordinates are very slightly different so that I cannot simply remove duplicates. As an example, let's say they look like this:
df1 = pd.DataFrame(data=None, columns=['x', 'y', 'z'])
df1.x = [104245, 252355, 547364, 135152]
df1.y = [842714, 135812, 425328, 124912]
df1.z = [125125, 547574, 364343, 346372]

df2 = pd.DataFrame(data=None, columns=['x', 'y', 'z'])
df2.x = [104230, 547298]
df2.y = [842498, 424989]
df2.z = [124976, 364001]

What I then want is for the first and second xyz-rows in df2 to remove the first and third row in df1. My idea was to create new columns with rounded values, compare those, and remove based on those. It would look something like this:
df1['id'] = np.linspace(0,len(df1)-1,len(df1))
df2['id'] = np.linspace(0,len(df2)-1,len(df2))

df3 = df1.round({'x': -3, 'y': -3, 'z': -3})
df4 = df2.round({'x': -3, 'y': -3, 'z': -3})

df5 = df3.merge(df4, on=['x', 'y', 'z'], how='inner')
df6 = df1[~df1.index.isin(df5.id_x)]

This works fine to remove some of the values, but often they round to different places. I was hoping with help if there is a better method to mask those values which are simply closest in all three coordinates. Maybe that it finds the closest xyz-pair between df1 and df2 and masks those pairs. If anyone has any ideas I would really appreciate it!


Answer:
You can use numpy broadcasting to consider the individual distances between the coordinates:
# convert DataFrames to numpy arrays
a1 = df1.to_numpy()
a2 = df2.to_numpy()

# define a distance below which the coordinates are considered equal
thresh = 500

# compute the distances, identify matches on all coordinates
matches = (abs(a1[:,None]-a2) <= thresh).all(axis=-1)

idx1, idx2 = np.where(matches)
# (array([0, 2]), array([0, 1]))

out = df1.drop(df1.index[idx1])

To consider the euclidean distance between the points (taking into account all coordinates simultaneously), use scipy.spatial.distance.cdist:
from scipy.spatial.distance import cdist

thresh = 1000

matches = cdist(a1, a2) <= thresh

idx1, idx2 = np.where(matches)

out = df1.drop(df1.index[idx1])

Output:
        x       y       z
1  252355  135812  547574
3  135152  124912  346372

removing the single point from df1 that is closest to each row of df2 and below a threshold
from scipy.spatial.distance import cdist

thresh = 1000

dist = cdist(a1, a2)

idx = np.argmin(dist, axis=0)

out = df1.drop(df1.index[idx[dist[idx, np.arange(len(a2))] <= thresh]])

If the distance doesn't matter and you only want to remove the closest point:
from scipy.spatial.distance import cdist

dist = cdist(a1, a2)

idx = np.argmin(dist, axis=0)

out = df1.drop(df1.index[idx])
Question: How to fix the line ending style (either CRLF or LF) in Python when written a text file?

I have the following little program in Python
from pathlib import Path
filename = Path("file.txt")
content = "line1\nline2\nline3\n"
with filename.open("w+", encoding="utf-8") as file:
    file.write(content)

After running it I get the following file (as expected)
line1
line2
line3

However, depending on where the program runs, line ending is different.
If I run it in Windows, I get CRLF line termination:
$ file -k file.txt
file.txt: ASCII text, with CRLF line terminators

If I run it in Linux, I get LF line termination:
$ file -k file.txt 
file.txt: ASCII text

So, I understand that Python is using the default from the system in which it runs, which is fine most of the times. However, in my case I'd like to fix the line ending style, no matter the system where I run the program.
How this could be done?


Answer:
It is possible to explicitly specify the string used for newlines using the newline parameter. It works the same with open() and pathlib.Path.open().
The snippet below will always use Linux line endings \n:
from pathlib import Path
filename = Path("file.txt")
content = "line1\nline2\nline3\n"
with filename.open("w+", encoding="utf-8", newline='\n') as file:
    file.write(content)

Setting newline='\r\n' will give Windows line endings and not setting it or setting newline=None (the default) will use the OS default.
Question: How to exclude linebreaks from a regex match in python?


How can I make the bellow regex exclude matches that span across lines?
import re
reg = re.compile(r'\b(apple)(?:\W+\w+){0,4}?\W+(tree|plant|garden)')
reg.findall('my\napple tree in the garden')
reg.findall('apple\ntree in the garden')

The first one should match, the second one should not.
(Now both matches...)


Answer:
Your \W matches newlines. To exclude them replace \W with [^\w\n]:
import re
reg = re.compile(r'\b(apple)(?:[^\n\w]+\w+){0,4}?[^\n\w]+(tree|plant|garden)')
print(reg.findall('my\napple tree in the garden'))
#  [('apple', 'tree')]
print(reg.findall('apple\ntree in the garden'))
#  []
Question: How to declare a constant datatype using a class property datatype in typescript?

I'm creating a nestjs API so I'm using classes to declare my entities for example
export class Customer {
id: number;
name: string;
}

So now I'm working in my Customer Controller and I would like to type an get query param as customer.id because I'm thinking if some day the customer id data type changes to string automatically making my controller query param as string too.
@GET()
getCustomerById(@Params('id', id: Customer.id)) {
return this.customerService.getCustomerById(id))
}

Is it possible? Thanks


Answer:
You can use TypeScript lookup types:
getCustomerById(@Params('id') id: Customer['id']) {}
Question: Different X Labels for Different Variables

I have two data frames t1 and t2. I want a seaborn plot where it plots side by side for every variable using the for loop. I was able to achieve this but I fail when I try to set the customized x labels. How do I incorporate set_xlabel in to the for loop?
data1 = {
    'var1': [1, 2, 3, 4],
    'var2': [20, 21, 19, 18],
    'var3': [5, 6, 7, 8]
    }

data2 = {
    'var1': [5. 2. 3. 5],
    'var2': [21, 18, 3, 11]
    'var3': [1, 9, 3, 6]
    }

  
t1 = pd.DataFrame(data1)
t2 = pd.DataFrame(data2)

xlabel_list = ["new_var1", "new_var2", "new_var3"]

def fun1(df1, df2, numvar, new_label):
    plt.tight_layout()

    fig, ax = plt.subplots(1, 2)
    sns.kdeplot(data = df1[numvar], linewidth = 3, ax=ax[0])
    sns.kdeplot(data = df2[numvar], linewidth = 3, ax=ax[1])

    ax[0].set_xlabel(new_label, weight='bold', size = 10)
    ax[1].set_xlabel(new_label, weight='bold', size = 10)    

for col in t1.columns: # how to incorporate the new_label parameter in the for loop along with col?
    fun1(df1 = t1, df2 = t2, numvar = col, new_label??)



Answer:
Use zip:
for col, new_label in zip(t1.columns, xlabels_list):