satoshi-2000/llms-suitable huggingface dataset:
Question: Clean column with different time records in powerbi

I have a Power BI table with a column called "hours". It can have different time records with  following different formats:
PT5H15M{YearMonthDayTime}, PT0S{YearMonthDayTime}, PT5H15M, PT10H, etc.
How can I clean them up so that the hours are represented as numbers, for example, PT5H15M{YearMonthDayTime} would be 5,25 and PT3H30M would be 3,5.
Can't find any easy way to filter the column since some rows have {YearMonthDayTime} ending and others doesn't. I don't want to transform every record manually.
Thanks already!


Answer:
You can achieve this in power query
Steps followed in Power Query

Extract Text between delimiters PT and H to populate Hours column. Change type to decimal
Extract Text between delimiters H and M to populate Minutes column. Change type to decimal
Replace Errors with zero. Replace all null values with zero.
Devide Minutes column by 60 to convert into hours.
Add Hours column in step 1) to hours column in step 4)
Convert result column type to Text
In result column: Replace . with , using Transform -> Replace Values

M code:
let
    Source = Excel.Workbook(File.Contents("C:\Ashok\Power BI\Stack Overflow\Data_02_jun2_2023.xlsx"), null, true),
    Data_Sheet = Source{[Item="Data",Kind="Sheet"]}[Data],
    #"Changed Type" = Table.TransformColumnTypes(Data_Sheet,{{"Column1", type text}}),
    #"Promoted Headers" = Table.PromoteHeaders(#"Changed Type", [PromoteAllScalars=true]),
    #"Changed Type1" = Table.TransformColumnTypes(#"Promoted Headers",{{"Hours", type text}}),
    #"Inserted Text Between Delimiters" = Table.AddColumn(#"Changed Type1", "Text Between Delimiters", each Text.BetweenDelimiters([Hours], "PT", "H"), type text),
    #"Inserted Text Between Delimiters1" = Table.AddColumn(#"Inserted Text Between Delimiters", "Text Between Delimiters.1", each Text.BetweenDelimiters([Hours], "H", "M"), type text),
    #"Changed Type2" = Table.TransformColumnTypes(#"Inserted Text Between Delimiters1",{{"Text Between Delimiters.1", type number}}),
    #"Inserted Division" = Table.AddColumn(#"Changed Type2", "Division", each [Text Between Delimiters.1] / 60, type number),
    #"Changed Type3" = Table.TransformColumnTypes(#"Inserted Division",{{"Text Between Delimiters", type number}}),
    #"Replaced Errors" = Table.ReplaceErrorValues(#"Changed Type3", {{"Text Between Delimiters", 0}}),
    #"xyz" = Table.TransformColumns(#"Replaced Errors", {{"Text Between Delimiters", each if _ is null then 0 else _}, 
                                                        {"Text Between Delimiters.1", each if _ is null then 0 else _}, {"Division", each if _ is null then 0 else _}}),
    #"Changed Type4" = Table.TransformColumnTypes(xyz,{{"Text Between Delimiters", type number}, {"Division", type number}}),
    #"Inserted Addition" = Table.AddColumn(#"Changed Type4", "Addition", each [Text Between Delimiters] + [Division], type number),
    #"Changed Type5" = Table.TransformColumnTypes(#"Inserted Addition",{{"Addition", type text}}),
    #"Replaced Value" = Table.ReplaceValue(#"Changed Type5",".",",",Replacer.ReplaceText,{"Addition"})
in
    #"Replaced Value"
Question: Write a Python program to remove duplicates from a list of integers, preserving order

So this was the question and I implemented it using python language but there is error in code and I am not able to fix it so please help. I am attaching code also.
def delete23(a):
    for i in range(0, len(a) - 1):
        for j in range(0, len(a) - 1):
            if a[i] == a[j] and i != j:
                a.pop(j)
    print(a)


a = [1, 2, 3, 4, 4, 1, 7]
print(len(a))
delete23(a)




Answer:
To filter duplicates while preserving the order, a set of seen values is used quite often. The logic is very simple: if a value is not present yet in seen, the value is encountered for the first time.
lst = [1, 2, 3, 4, 4, 1, 7]

seen = set()
lst2 = []
for x in lst:
    if x not in seen:
        seen.add(x)
        lst2.append(x)
    
print(lst2)

With a little trick, this can be shortened:
seen = set()
lst2 = [(seen.add(x) or x) for x in lst if x not in seen]

print(lst2)

To explain the expression inside (...): set.add returns None and (None or VALUE) is evaluated to VALUE

Warning: ALL solutions using a set or a dict require that all values are hashable.
Question: Apply a weighting to a 4 parameter regression curvefit

The below code generates a plot and 4PL curve fit, but the fit is poor at lower values. This error can usually be addressed by ading a 1/y^2 weighting, but I dont know how to do it in this instance. Adding sigma=1/Y_data**2 to the fit just makes it worse.
from scipy.optimize import curve_fit
import matplotlib.pyplot as plt

def fourPL(x, A, B, C, D):
    return ((A-D) / (1.0 + np.power(x / C, B))) + D

X_data = np.array([700,200,44,11,3,0.7,0.2,0])
Y_data = np.array([600000,140000,30000,8000,2100,800,500,60])


popt, pcov = curve_fit(fourPL, X_data, Y_data)

fig, ax = plt.subplots()    
ax.scatter(X_data, Y_data, label='Data')
X_curve = np.linspace(min(X_data[np.nonzero(X_data)]), max(X_data), 5000)
Y_curve = fourPL(X_curve, *popt)
ax.plot(X_curve, Y_curve)

ax.set_xscale('log')
ax.set_yscale('log')

plt.show()




Answer:
Don't add inverse square weights; fit in the log domain. Always add bounds. And in this case, curve_fit doesn't do a very good job; consider instead minimize.
import numpy as np
from scipy.optimize import curve_fit, minimize
import matplotlib.pyplot as plt


def fourPL(x: np.ndarray, a: float, b: float, c: float, d: float) -> np.ndarray:
    return (a - d)/(1 + (x / c)**b) + d


def estimated(x: np.ndarray, a: float, b: float, c: float, d: float) -> np.ndarray:
    return np.log(fourPL(x, a, b, c, d))


def sqerror(abcd: np.ndarray) -> float:
    y = np.log(fourPL(x_data, *abcd)) - np.log(y_data)
    return y.dot(y)


x_data = np.array([700, 200, 44, 11, 3, 0.7, 0.2, 0])
y_data = np.array([600000, 140000, 30000, 8000, 2100, 800, 500, 60])
guess = (500, 1.05, 1e6, 1e9)
bounds = np.array((
    (1, 0.1, 1, 0),
    (np.inf, 10, np.inf, np.inf),
))

popt, _ = curve_fit(
    f=estimated, xdata=x_data, ydata=np.log(y_data), p0=guess,
    bounds=bounds,
)
print('popt:', popt)
result = minimize(
    fun=sqerror, x0=guess, bounds=bounds.T, tol=1e-9,
)
assert result.success
print('minimize x:', result.x)

x_curve = 10**np.linspace(-1, 3, 1000)

fig, ax = plt.subplots()
ax.scatter(x_data, y_data, label='Data')
ax.plot(x_curve, fourPL(x_curve, *popt), label='curve_fit')
ax.plot(x_curve, fourPL(x_curve, *result.x), label='minimize')
ax.plot(x_curve, fourPL(x_curve, *guess), label='guess')
ax.set_xscale('log')
ax.set_yscale('log')
ax.legend()
plt.show()
Question: How can I correctly check the type of a Redis key in Lua scripting?

Correct way to check redis key's type in lua script.
I'm trying to wrap my head around redis lua scripting and I can't find the correct way to check what type the key has.
Here is what I've tried:
127.0.0.1:6379> SET test_key test_value
OK
127.0.0.1:6379> GET test_key
"test_value"
127.0.0.1:6379> EVAL 'local type = redis.call("TYPE", KEYS[1]); return type' 1 test_key
string

So I see that the type = "string", but:
127.0.0.1:6379> EVAL 'local type = redis.call("TYPE", KEYS[1]); local res; if type == string then res = "ok" else res = "not ok" end return res' 1 test_key
"not ok"
127.0.0.1:6379> EVAL 'local type = redis.call("TYPE", KEYS[1]); local res; if type == string then res = "ok" else res = "not ok" end return res' 1 "test_key"
"not ok"
127.0.0.1:6379> EVAL 'local type = redis.call("TYPE", KEYS[1]); local res; if type == "string" then res = "ok" else res = "not ok" end return res' 1 "test_key"
"not ok"
127.0.0.1:6379> EVAL 'local type = redis.call("TYPE", KEYS[1]); local res; if type == "string" then res = "ok" else res = "not ok" end return res' 1 test_key
"not ok"



Answer:
I've found the answer here: Using the TYPE command inside a Redis / Lua Script
Short answer is that in lua scripts redis.call("TYPE", key) returns not string but lua table with key "ok" which holds string value of the type.
So to check the type of the key you should compare like this:
if redis.call("TYPE", key)["ok"] == "string"

for example:
127.0.0.1:6379> EVAL 'local type = redis.call("TYPE", KEYS[1])["ok"]; local res; if type == "string" then res = "ok" else res = "not ok" end return res' 1 test_key
"ok"
Question: How can I prioritize dimensions in a Redis vector similarity search?

I am currently using Redis as a vector database and was able to get a similarity search going with 3 dimensions (the dimensions being latitude, longitude, and timestamp). The similarity search is working but I would like to weigh certain dimensions differently when conducting the search. Namely, I would like the similarity search to prioritize the timestamp dimension when conducting the search.
How would I go about this? Redis does not seem to have any built-in feature that does this.
I turn each set of lat, long, and time coordinates into bytes that can be put into the vector database with the following code. Note that vector_dict stores all the sets of lat, long, and timestamp:
p = client.pipeline(transaction=False)
for index in data:
        # create hash key
        key = keys[index]

        # create hash values
        item_metadata = data[index] # copy all metadata
        item_key_vector = np.array(vector_dict[index]).astype(np.float32).tobytes() # convert vector to bytes
        p.hset(key, mapping=item_metadata) # add item to redis using hash key and metadata

I then conduct the similarity search using the HNSW index here:
def create_hnsw_index(redis_conn, vector_field_name, number_of_vectors, vector_dimensions=3, distance_metric='L2', M=100, EF=100):
    redis_conn.ft().create_index([
        VectorField(vector_field_name, "HNSW", {"TYPE": "FLOAT32", "DIM": vector_dimensions, "DISTANCE_METRIC": distance_metric, "INITIAL_CAP": number_of_vectors, "M": M, "EF_CONSTRUCTION": EF})
    ])

I talked with others and they said it is a math problem that deals with vector normalization. I'm unsure how to get started with this though in code and would like some guidance.


Answer:
You can re-weight the vector to make certain dimensions longer than others. You're using an L2 distance metric. That uses the standard Pythagorean theorem to calculate distance:
dist = sqrt((x1-x2)**2 + (y1-y2)**2 + (z1-z2)**2)

Imagine you multiplied every Y value, in both your query and your database, by 10. That would also multiply the difference between Y values by a factor of 10.
The new distance function would effectively be this:
dist = sqrt((x1-x2)**2 + (10*(y1-y2))**2 + (z1-z2)**2)

dist = sqrt((x1-x2)**2 + 100*(y1-y2)**2 + (z1-z2)**2)

...which makes the Y dimension matter 100 times more than the other dimensions.
So if you want the dimension in position 2 to matter more, you could do this:
item_key_vector = np.array(vector_dict[index])
item_key_vector[2] *= 10
item_key_vector_bytes = item_key_vector.astype(np.float32).tobytes()

The specific amount to multiply by depends on how much you want the timestamp to matter. Remember that you need to multiply your query vector by the same amount.
Question: Why pybind11 can not recognize PyObject* as a Python object and how to fix it?

I am trying to build a library using this C++ code:
#include <pybind11/pybind11.h>

namespace py = pybind11;

PyObject* func() {
    return Py_BuildValue("iii", 1, 2, 3);
}

PYBIND11_MODULE(example, m) {
    m.def("func", &func);
}

But when I tried to run this Python 3 code:
import example
print(example.func())

It gives me following error:
TypeError: Unregistered type : _object

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
TypeError: Unable to convert function return value to a Python type! The signature was
        () -> _object

Why it is happening and how to fix it?


Answer:
So, first off, what are you trying to do with your function? Making a raw python API call (Py_BuildValue, https://docs.python.org/3/c-api/arg.html#c.Py_BuildValue) is strongly discouraged, that is why you are using PyBind11, to handle that for you.
What are you trying to do with this line?
Py_BuildValue("iii", 1, 2, 3)

It looks like you are trying to return a tuple of 3 ints. Perhaps something like this would work instead:
py::tuple func() {
    return py::make_tuple(1, 2, 3);
}

With that said, I think the error is from Python not understanding what a PyObject* is. So you will need to expose the PyObject to Python using a py::class. I'm not sure if that makes sense though, I would need to test this.
From the docs I linked, it looks like Py_BuildValue might return a tuple. In which case I can suggest wrapping the return value in a py::tuple.
Question: How to get time from datetime into a string/number for a colormap

I'm trying to shade a scatter plot based on time of day, but my data is in datetime format. Getting either time of day or hours past 00:00 would work. I tried to get just the time from datetime, but I got the following error.

TypeError: float() argument must be a string or a number, not 'datetime.time'

Ideally, I'll have a scatterplot shaded based on time of day.
I initially tried this (also added .values to the end of the dt.time to see if it would help. It didn't).
x = dfbelow[1:].wind_speed.values
y = above_aligned["WS"].values
plt.scatter(x, y, s=20, c=above_aligned["timestamp"].dt.time, cmap='YlOrBr')

plt.xlabel("Below Canopy Wind Speed (m/s)")
plt.ylabel("Above Canopy Wind Speed (m/s)")
plt.title("Above vs. Below Canopy Wind Speeds (m/s)")
plt.colorbar(label="Below Canopy Wind Direction")
plt.show

But it understandably can't shade based off of a datetime form.


Answer:
I figured it out and it was actually very simple. Instead of above_aligned["timestamp"].dt.time, I just used above_aligned["timestamp"].dt.hour.
Question: AWS ECS Task on private subnet connectivity

Is it possible to access a public domain e.g. foo.bar.com from an AWS ECS task running on a private subnet? It is convenient for me that the task is running on a private subnet since it can be easily accessible from other ECS tasks running on the same private subnet (same AZ, VPC and region).
I am reading different contradicting opinions:

Some say that this is not possible and I must configure a NAT Gateway or NAT instance
Others say that this is possible as long as you specify the right outbound rules for your security groups. But outbound rules can only be configured using IP ranges and not specific domains.

What is actually the case here?


Answer:
You have to create a NAT Gateway and add a route in the route table for your subnet. Without a NAT Gateway (or NAT instance, but those are considered obsolete), you cannot connect to the public internet from your private subnets.
Question: PHP function file_get_contents($url) returns special characters

I am trying to retrieve the meta data for a given links (url). I have implemented the following steps:
$url = "url is here";
$html = file_get_contents($url);
$crawler = new Crawler($html); // Symfony library
$description = $crawler->filterXPath("//meta[@name='description']")->extract(['content']);

Doing so, I manage to retrieve the meta data for some urls but not for all.
Some urls, the file_get_contents($url) function returns special characters like (x1F‹\x08\x00\x00\x00\x00\x00\x04\x03ì½}{ãÆ‘/ú÷øSÀœ'\x1E)! ‘z§¬qlÇI..........) that is why I could not retrieve the meta data.
Notice that, I am using the same website for $url values but passing different slugs (different blog urls like https://www.example.com/blog-1).
Attempts:

I used these functions  mb_convert_encoding and mb_detect_encoding
I made sure all urls I have passed are accessible through the browser.

Any thought, why I am getting special characters when I am calling file_get_contents function, and some time getting correct html format?



Answer:
I have solved the issue by adding the following parameters to file_get_contents functions:
private const EMBED_URL_APPEND = '?tab=3&object=%s&type=subgroup';
      
private const EMBED_URL_ENCODE= 'CM_949A11_1534_1603_DAG_DST_50_ÖVRIGT_1_1';
            
$urlEncoded= sprintf($url.self::EMBED_URL_APPEND, rawurlencode(self::EMBED_URL_ENCODE));
            
$html =  file_get_contents($urlEncoded);
Question: Change x or y position of density plot

I have data plotted as points and would like to add density plots to the graph. The marginal plot solutions from ggExtra or other packages are not giving the freedom that I'd like and so want to generate the density plot at the same time as the ggplot.
df = data.frame(x = rnorm(50, mean = 10), 
                y = runif(50, min = 10, max = 20), 
                id = rep(LETTERS[1:5], each = 10))
ggppp = ggplot(data = df, mapping = aes(x, y, color = id)) + 
  geom_point() + theme_bw()

ggppp + 
  geom_density(mapping = aes(y = y, 
                             col = id),
               inherit.aes = FALSE, bounds = c(-Inf, Inf)) +
  geom_density(mapping = aes(x = x,
                             col = id),
               inherit.aes = FALSE, ) 


Is there a way to move the density plots to other values of x or y position (like moving the density lines to the tip of the arrow in the image below)?


Answer:
you can shift the position with position_nudge:
## using your example objects:
ggppp + 
  geom_density(mapping = aes(y = y , col = id),
               position = position_nudge(x = 12),
               inherit.aes = FALSE
               ) +
  geom_density(mapping = aes(x = x, col = id),
               position = position_nudge(y = 20),
               inherit.aes = FALSE
)
Question: Why am I unable to retrieve a Xero item by identifier with a valid access token from Postman?

xero developer api not authorizing
i generated the access_token from the endpoint postman screen shot for access token when i try to get xero item i am getting screen shot for item end point this endpoint should give the item with identifier 96d14376-4b75-4b4a-8fd3-b1caab075ab3 in the response also when i try this one in xero api explorer after login its working fine


Answer:
Looking at the logs relating to the instance id in the error screen shot, the
access token does not include the accounting.settings scope.
Please can you go through the OAuth 2.0 process from the very beginning, making sure the scope is in the authorisation call.
When you add a new scope to a call you need to go through the whole authorisation process from scratch to update the access token.
When you get a new access token you can decode it to check the scopes before you use it to make sure that you have the scopes you need. You can use jwt.io to check this if you wish
Question: Call Different Method in reactive pipeline

I have a use case where I have to call two different methods in a reactive pipeline Java 8 on a post-API call.
1st Method:
will insert data in a master table and will return the pk of that table insertion.
2nd Method:
this method will insert data in the mapping table which will use the pk received from 1st method.
I try to do that Mono.zip, but that did not work as Zip is calling both methods simultaneously and is not able to pass method 1 output to 2nd method input.


Answer:
You can easily do it using the map or flatMap operator, depending on what type of repository you have - reactive or not reactive.
Here is examples for both cases:
public class YourService {
    private final YourRepository repository;
    private final YourReactiveRepository reactiveRepository;

    void doAction() {
        Mono.fromCallable(() -> repository.saveToMainTable("main data"))
            .map(mainTableId -> repository.saveToSecondaryTable(mainTableId, "secondary data"))
            .subscribeOn(Schedulers.boundedElastic())
            .subscribe();
    }

    void doActionWithReactiveRepository() {
        reactiveRepository.saveToMainTable("main data")
            .flatMap(mainTableId -> reactiveRepository.saveToSecondaryTable(mainTableId, "secondary data"))
            .subscribe();
    }

    interface YourRepository {
        int saveToMainTable(String someData);
        boolean saveToSecondaryTable(int mainTableId, String someData);
    }

    interface YourReactiveRepository {
        Mono<Integer> saveToMainTable(String someData);
        Mono<Boolean> saveToSecondaryTable(int mainTableId, String someData);
    }
}

You can read more about map here and about flatMap here
Question: Sort pandas dataframe columns on second row order

I need to sort a dataframe based on the order of the second row.  For example:
import pandas as pd

data = {'1a': ['C', 3, 1], '2b': ['B', 2, 3], '3c': ['A', 5, 2]}
df = pd.DataFrame(data)
df

Output:
  1a 2b 3c
0  C  B  A
1  3  2  5
2  1  3  2

Desired output:
  3c 2b 1a
0  A  B  C
1  5  2  3
2  2  3  1

So the columns have been order based on the zero index row, on the A, B, C.
Have tried many sorting options without success.
Having a quick way to accomplish this would be beneficial, but having granular control to both order the elements and move a specific column to the first position would be even better.  For example move "C" to the first column.
Something like make a list, sort, move and reorder on list.
mylist = ['B', 'A', 'C']
mylist.sort()
mylist.insert(0, mylist.pop(mylist.index('C')))

Then sorting the dataframe on ['C', 'A', 'B'] outputting
  1a 3c 2b
0  C  A  B
1  3  5  2
2  1  2  3



Answer:
With the help of pyjedy and Stingher, I was able to resolve this issue. One of the problems was due to my input. The input consisted of lists instead of dictionaries, so I needed to transform it. As a result, I had indexes for rows and across the top for columns. Consequently, selecting elements from the list required obtaining the index.
import pandas as pd

def search_list_for_pattern(lst, pattern):
    for idx, item in enumerate(lst):
        if pattern in item:
            break
    return idx

data = [['1a', 'B', 2, 3], ['2b', 'C', 3, 1], ['3c', 'A', 5, 2]]
df = pd.DataFrame(data).transpose()
print(df)

#     0   1   2
# 0  1a  2b  3c
# 1   B   C   A
# 2   2   3   5
# 3   3   1   2

# Get the second row and convert it to a list
second_row = df.iloc[1, :].tolist()
print(second_row)

# ['B', 'C', 'A']

# Find the index of the column you want to move to the first position
target_column = search_list_for_pattern(second_row, "C")
print(target_column)

# 1

# Sort the list
sorted_columns = sorted(range(len(second_row)), key=lambda k: second_row[k])
print(sorted_columns)

# [2, 0, 1]

# Move the target column to the first position
sorted_columns.remove(df.columns.get_loc(target_column))
sorted_columns.insert(0, df.columns.get_loc(target_column))

# Reorder the columns of the DataFrame based on the sorted list
df = df.iloc[:, sorted_columns]
print(df)

#     1   2   0
# 0  2b  3c  1a
# 1   C   A   B
# 2   3   5   2
# 3   1   2   3

df.to_excel('ordered.xlsx', sheet_name='Sheet1', index=False, header=False)