satoshi-2000/llms-suitable huggingface dataset:
Question: Harbor registry proxy cache vs replication

I'm new to Harbor registry. I was asked to propose an architecture for harbor in my company. I proposed at first to use an architecture based on proxy cache. But the CISO refused to use proxy cache for the entreprise without saying why. I proposed anoter architecture based on replication. We validate some base images that are pulled from public registries and pushed into our harbor registry ( One active harbor that pulls the images from internet and another passive harbor for high avalibility + 4 other harbors that leaves in special network zones (they get the images form the master harbor)).
The question is why the ciso refused the use of proxy cache ? is there any drawbacks for using it ? what are the security risks that can appear using the harbor proxy cache vs replication ? I cant find in the internet clear informations about this question. It seems that the majority is using proxy cache.
Thank you!


Answer:
At this stage one can only speculate, about the unprofessional behavior of not explaining the reasons and also for not asking.
Regarding Harbor proxy and replication, the main difference between both option is the difference of threat surface and its control.
Proxy

Passive, forwards requests upstream if not found locally.
No control,

Replication

Active, explicitly specify the images you want to copy from upstream
Full control
Question: Reshaping a Dataframe with repeating column names

I have data that looks like this:
    dataframe_1:                
    week    SITE        LAL SITE     LAL
0   1   BARTON CHAPEL   1.1 PENASCAL I  1
1   2   BARTON CHAPEL   1.1 PENASCAL I  1
2   3   BARTON CHAPEL   1.1 PENASCAL I  1

And, i need the final dataframe to look like this:
    dataframe_2:        
    week    SITE    LAL
0   1   BARTON CHAPEL   1.1
1   2   BARTON CHAPEL   1.1
2   3   BARTON CHAPEL   1.1
3   1   PENASCAL I  1
4   2   PENASCAL I  1
5   3   PENASCAL I  1

I've tried using 'melt' but I cannot get the desire result. Perhaps I'm using the wrong approach?
thank you,


Answer:
Not a very generalizable solution, but will work on your example:
df.groupby('week').apply( lambda _df : pd.concat((_df.iloc[:,1:3], _df.iloc[:,3:5]))).reset_index('week')

it groups by week and then reshapes with column selection + concatenation. Removing a superfluous index column in the end.
Question: How to set a timeout for a popup and close if user clicks elsewhere?

I created a popup that appears when I click a button, but to make it disappear I have to click again. Is there a way to set a timer and make it disappear?
Function:
// When the user clicks on div, open the popup
function myFunction() {
  var popup = document.getElementById("myPopup");
  popup.classList.toggle("show");
}

Style:
.popuptext {
  display: none;
}
.popuptext.show {
  display: block;
}

The HTML:
<div class="popup" onclick="myFunction()">Click me to toggle the popup!
  <span class="popuptext" id="myPopup">A Simple Popup!</span>
</div>

I need the popup to close after 10 seconds OR when the user clicks somewhere else.
I edited the code to below and it does close after 10 seconds, how to achieve the second part (close when user clicks somewhere else):
function myFunction() {
var popup = document.getElementById("myPopup");
popup.classList.toggle("show");
if(popup.classList.contains("show"))
    setTimeout(() => popup.classList.remove("show"), 10000)
}



Answer:
To do this you need to:

Define a function, hide() that hides the popup.
Add an mousedown event listener to the whole document that invokes hide

Within hide, ensure that the click event's target is not contained in the popup.


Set up the timeout to call hide
Important: Have hide clear the created timeout and remove the listener that was added.



function myFunction() {
  var popup = document.getElementById("myPopup");
  popup.classList.add('show')
  
  let timeout;
  
  function hide(e) {
    if (popup.contains(e.target)) return;

    popup.classList.remove("show");
    document.removeEventListener('mousedown', hide);
    clearTimeout(timeout)
  }
  document.addEventListener('mousedown', hide)
  timeout = setTimeout(hide, 10000)
}
.popuptext {
  display: none;
}

.popuptext.show {
  display: block;
}
<div class="popup" onclick="myFunction()">Click me to toggle the popup!
  <span class="popuptext" id="myPopup">A Simple Popup!</span>
</div>
Question: Index of .iloc API in Pandas

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 400 entries, 0 to 399
Data columns (total 11 columns):
 #   Column       Non-Null Count  Dtype 
---  ------       --------------  ----- 
 0   CompPrice    400 non-null    int64 
 1   Income       400 non-null    int64 
 2   Advertising  400 non-null    int64 
 3   Population   400 non-null    int64 
 4   Price        400 non-null    int64 
 5   ShelveLoc    400 non-null    object
 6   Age          400 non-null    int64 
 7   Education    400 non-null    int64 
 8   Urban        400 non-null    object
 9   US           400 non-null    object
 10  HighSales    400 non-null    object
dtypes: int64(7), object(4)
memory usage: 34.5+ KB

As shown in the info() result above, there are 11 columns indexed from 0 to 10 in my dataset, DF. Now, I would like to extract only the first 10 columns (that are the columns with the indices 0 to 9). However, when I try to use the code below:
DF.iloc[:, 0:9]

It returns only the first 9 columns (that is, from CompPrice to Urban).
In this case, I need to change my code to:
DF.iloc[:, 0:10]

to get what I actually want (that is, from CompPrice to US).
I'm really confused by iloc() indices. Why it requires '10' instead '9' but starts with the index '0'. The starting and ending indices are not consistent.


Answer:
What you are observing is the standard functionality of pandas. If you look in the documentation, you can find the definition. This is intended and logical, as Python lists function the same way. As per the docs:

.iloc is primarily integer position based (from 0 to length-1 of the axis), but may also be used with a boolean array. .iloc will raise IndexError if a requested indexer is out-of-bounds, except slice indexers which allow out-of-bounds indexing. (this conforms with Python/NumPy slice semantics).
Question: Ruby IPAddr class accepting wrong IPv6 address string

We are middle of upgrading ruby versions v2.7.3 -> v3.1.3
One of our test cases are failing related to valid ipv6 address string, check the following
# ruby 2.7.3
IPAddr.new('fe80::85e:7530:69ec:9074%en0').ipv6?
=> IPAddr::InvalidAddressError (invalid address: fe80::85e:7530:69ec:9074%en0)

# ruby 3.1.3

IPAddr.new('fe80::85e:7530:69ec:9074%en0').ipv6?
=> true

Is it really a bug or am I missing something? Please help..


Answer:

Is it really a bug or am I missing something?

This used to be an issue in the ipaddr default gem up to version 1.2.2 which was fixed in version 1.2.3 in order to be fully compliant with RFC 4007 and RFC 6874. Version 1.2.3 of the ipaddr default gem was shipped as part of Ruby 3.1.0.
So, you are correct. This is a bug (although opinions differ on that) which was fixed in ipaddr 1.2.3 / Ruby 3.1.0.
Question: Why does pandas `date_range` rounds up to the next month?

When using pandas.date_range with start date, frequency, and periods the date range rounds up when using the start date as the last day of a month.
It seems like a silent edge case bug. If it's not a bug, any idea why it does that?
For example
import pandas as pd

start_date = pd.Timestamp(2023, 5, 31)
date_range = pd.date_range(start=start_date, freq="MS", periods=6)

results in
DatetimeIndex(['2023-06-01', '2023-07-01', '2023-08-01', '2023-09-01',
               '2023-10-01', '2023-11-01'],
              dtype='datetime64[ns]', freq='MS')

From the documentation, I'd expect it to start in May and end in October:
DatetimeIndex(['2023-05-01', '2023-06-01', '2023-07-01', '2023-08-01', '2023-09-01',
               '2023-10-01'],
              dtype='datetime64[ns]', freq='MS')

I thought it had to do with the inclusive argument but that's not the reason either.


Answer:
pd.date_range is to generate a range of date between start and end. 2023-05-01 is less than start date 2023-05-31, it will never reach it. To do what you want, you can replace the day of pd.Timestamp by 1.
start_date = pd.Timestamp(2023, 5, 31)
date_range = pd.date_range(start=start_date.replace(day=1), freq="MS", periods=6)

print(date_range)

DatetimeIndex(['2023-05-01', '2023-06-01', '2023-07-01', '2023-08-01',
               '2023-09-01', '2023-10-01'],
              dtype='datetime64[ns]', freq='MS')
Question: What is the bug in my Persistent UTM Code?

I am using the following code to persist utms across my website. However i notice that its adding a question mark to links even without the UTM parameters. Can someone help me figure out what in this code needs to change. It should only be trying to add UTM parameters to the links if there is one present in the URL.
 <script>
(function() {
  var domainsToDecorate = [
          'example.com'
      ],
      queryParams = [
          'utm_medium', //add or remove query parameters you want to transfer
          'utm_source',
          'utm_campaign',
          'utm_content',
          'utm_term'
      ]
  // do not edit anything below this line
  var links = document.querySelectorAll('a'); 

// check if links contain domain from the domainsToDecorate array and then decorates
  for (var linkIndex = 0; linkIndex < links.length; linkIndex++) {
      for (var domainIndex = 0; domainIndex < domainsToDecorate.length; domainIndex++) { 
          if (links[linkIndex].href.indexOf(domainsToDecorate[domainIndex]) > -1 && links[linkIndex].href.indexOf("#") === -1) {
              links[linkIndex].href = decorateUrl(links[linkIndex].href);
          }
      }
  }
// decorates the URL with query params
  function decorateUrl(urlToDecorate) {
      urlToDecorate = (urlToDecorate.indexOf('?') === -1) ? urlToDecorate + '?' : urlToDecorate + '&';
      var collectedQueryParams = [];
      for (var queryIndex = 0; queryIndex < queryParams.length; queryIndex++) {
          if (getQueryParam(queryParams[queryIndex])) {
              collectedQueryParams.push(queryParams[queryIndex] + '=' + getQueryParam(queryParams[queryIndex]))
          }
      }
      return urlToDecorate + collectedQueryParams.join('&');
  }


  // a function that retrieves the value of a query parameter
  function getQueryParam(name) {
      if (name = (new RegExp('[?&]' + encodeURIComponent(name) + '=([^&]*)')).exec(window.location.search))
          return decodeURIComponent(name[1]);
  }

})();
</script>



Answer:
In decorateUrl you are adding the ? if there is not one
urlToDecorate = (urlToDecorate.indexOf('?') === -1) ? urlToDecorate + '?' : urlToDecorate + '&';

I would suggest you may only want to do this if collectedQueryParams contains any elements
 function decorateUrl(urlToDecorate) {
      var collectedQueryParams = [];
      for (var queryIndex = 0; queryIndex < queryParams.length; queryIndex++) {
          if (getQueryParam(queryParams[queryIndex])) {
              collectedQueryParams.push(queryParams[queryIndex] + '=' + getQueryParam(queryParams[queryIndex]))
          }
      }

      if(collectedQueryParams.length == 0){
        return urlToDecorate;
      }

      //only add the ? if we have params AND if there isn't already one
      urlToDecorate = (urlToDecorate.indexOf('?') === -1) ? urlToDecorate + '?' : urlToDecorate + '&';
      return urlToDecorate + collectedQueryParams.join('&');
  }
Question: How can I convert ISCII encoding to unicode for Gujarati language in Python 3?

I have some Gujarati string but its in ISCII encoding, so python throughing error (SyntaxError: invalid decimal literal).
string = TFH[TZDF\ I]GF.8[0 G[Xg;

line 1
    string = TFH[TZDF\ I]GF.8[0 G[Xg;
                      ^
SyntaxError: unexpected character after line continuation character

I was tried byte encoding too, but its not giving output like ISCII encoding.
I am trying this to make ISCII into unicode for Gujarati language.
I have ISCII based font and character map data also.
ISCII input string: TFH[TZDF\ I]GF.8[0 G[Xg;
Desired unicode output: તાજેતરમાં યુનાઇટેડ નેશન્સ (Typed using gujarati phonetic keyboard)


Answer:
If you just want to write the string literal, for me, just writing print("તાજેતરમાં યુનાઇટેડ નેશન્સ") worked.
Or you could write:
characters = [2724, 2750, 2716, 2759, 2724, 2736, 2734, 2750, 2690, 32, 2735, 2753, 2728, 2750, 2695, 2719, 2759, 2721, 32, 2728, 2759, 2742, 2728, 2765, 2744]
string = str()
for c in characters:
        string += chr(c)

Maybe you have a look at this conversion script:
https://gist.github.com/pathumego/81672787807c23f19518c622d9e7ebb8
Question: Extracting vcf columns substring with awk

I have vcf file like this:
##bcftools_annotateVersion=1.3.1+htslib-1.3.1
##bcftools_annotateCommand=annotate 
#CHROM  POS ID  REF ALT QUAL    FILTER  INFO    FORMAT  HG005
chr1    817186  rs3094315   G   A   50  PASS    platforms=2;platformnames=Illumina,CG;datasets=3;datasetnames=HiSeq250x250,CGnormal,HiSeqMatePair;callsets=5;callsetnames=HiSeq250x250Sentieon,CGnormal,HiSeq250x250freebayes,HiSeqMatePairSentieon,HiSeqMatePairfreebayes;datasetsmissingcall=IonExome,SolidSE75bp;callable=CS_HiSeq250x250Sentieon_callable,CS_CGnormal_callable,CS_HiSeq250x250freebayes_callable;AN=2;AF=1;AC=2 GT:PS:DP:ADALL:AD:GQ    1/1:.:809:0,363:78,428:237
chr1    817341  rs3131972   A   G   50  PASS    platforms=3;platformnames=Illumina,CG,Solid;datasets=4;datasetnames=HiSeq250x250,CGnormal,HiSeqMatePair,SolidSE75bp;callsets=6;callsetnames=HiSeq250x250Sentieon,CGnormal,HiSeq250x250freebayes,HiSeqMatePairSentieon,HiSeqMatePairfreebayes,SolidSE75GATKHC;datasetsmissingcall=IonExome;callable=CS_HiSeq250x250Sentieon_callable,CS_CGnormal_callable,CS_HiSeq250x250freebayes_callable;AN=2;AF=1;AC=2   GT:PS:DP:ADALL:AD:GQ    1/1:.:732:1,330:99,391:302

I need to extract ID column and  AN from INFO column to get:
ID         INFO
rs3094315   2
rs3131972   2

I'm trying something like this awk '/^[^#]/ { print $3, gsub(/^[^AN=])/,"",$8)}' file.vcf, but still not getting the desired result.


Answer:
You can try this awk:
awk 'BEGIN{OFS="\t"}
/^##/{next}
/^#/{print $3,$8; next}
{
    split($8,a,";")
    for(i=1;i<=length(a);i++) if (a[i]~/^AN=/) {sub(/^AN=/,"",a[i]); break}
    printf "%s%s%s\n", $3, OFS, a[i]
}
' file

With the example, prints:
ID  INFO
rs3094315   2
rs3131972   2
Question: ggplot2: Date range bar graph

I am trying to build a date range bar graph usins ggplot2 (R) in the spirit of:

I have followed a thread but I am completely unable to reproduce the results with dates.
If I understood it correctly, for each "id", the bar length is determined by the smallest and largest "value" in the database.
Here is a minimally working example of my data:
# Example dataframe
DF <- data.frame(Name = as.factor(c("1.Project1", "2.Project2", "3.Project3", "4.Project4")),
                 CreationTime = as.POSIXct(c("2019-12-10 13:22:20", "2019-12-17 12:25:48", "2020-01-02 13:02:57", "2020-01-14 08:37:10")),
                 LastActivity = as.POSIXct(c("2019-12-17 10:42:17 ", "2020-01-02 13:27:10", "2021-02-11 11:32:45", "2023-05-03 07:41:38")),
                 Status = as.factor(c("Prod", "Prod", "Dev", "Complete")))

# From wide to long
DFGather <- DF %>% tidyr::gather(key="Time", value="Value", 2:3)

# Generate plot
ggplot2::ggplot(DFGather, aes(x = Value, y = Name, fill = Status)) +
  ggplot2::geom_col() +
  ggplot2::coord_cartesian(xlim = c(min(DFGather$Value),max(DFGather$Value))) +
  ggplot2::scale_x_datetime(date_breaks = "3 months", labels = scales::label_date_short())

I have also tried converting POSIXct dates to integers but it didn't change my output:
DFGather$Value <- as.integer(format(DFGather$Value,"%Y%m%d"))

Thanks for the support,
C.


Answer:
A quick and dirty approach using geom_segment.

ggplot2::ggplot(DF, ggplot2::aes(x = CreationTime, xend = LastActivity, y = Name, yend = Name, colour = Status)) +
  ggplot2::geom_segment(linewidth = 15) +
  ggplot2::coord_cartesian(xlim = c(min(DFGather$Value),max(DFGather$Value))) +
  ggplot2::scale_x_datetime(date_breaks = "3 months", labels = scales::label_date_short())


Created on 2023-06-01 with reprex v2.0.2
Question: Spark is unable to handle a particular date format

I am trying to cast multiple date formats from string type field using Pyspark. When I am using below date format it is working fine.
def custom_to_date(col):
    formats = ("MM/dd/yyyy", "yyyy-MM-dd", "dd/MM/yyyy", "MM/yy","dd/M/yyyy")
    return coalesce(*[to_date(col, f) for f in formats])
    
from pyspark.sql.functions import coalesce, to_date
df = spark.createDataFrame([(1, "01/22/2010"), (2, "2018-12-01")], ("id", "dt"))
df.withColumn("pdt", custom_to_date("dt")).show()

Above code gives the correct output.
But when I use the month in single digit as below, the code fails.
df = spark.createDataFrame([(1, "01/22/2010"), (2, "2018-12-1"),(3,"24/7/2006")], ("id", "dt"))

I got the below error message.
org.apache.spark.SparkException:
  Job aborted due to stage failure:
    Task 2 in stage 2.0 failed 4 times, most recent failure: 
      Lost task 2.3 in stage 2.0 (TID 10) (10.13.82.55 executor 0): 
        org.apache.spark.SparkUpgradeException: 
        [INCONSISTENT_BEHAVIOR_CROSS_VERSION.PARSE_DATETIME_BY_NEW_PARSER] 
        You may get a different result due to the upgrading to Spark >= 3.0:



Answer:
Adding an answer since the comments and others answer doesn't cover the behaviour. The solution is not to add new formats. Since the formats itself can be better defined.
with spark 3.0 M supports 01, 1. January, Jan.
So you don't need MM
spark reference - https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html
def custom_to_date(col):
    formats = ("M/d/yyyy", "yyyy-M-d", "d/M/y", "M/y")
    return coalesce(*[to_date(col, f) for f in formats])
    
from pyspark.sql.functions import coalesce, to_date
df = spark.createDataFrame([(1, "01/22/2010"), (2, "2018-12-1"),(3,"12/2023")], ("id", "dt"))
df.withColumn("pdt", custom_to_date("dt")).show()

Results -

Alternatively, if you want legacy behavior then you can use
spark.conf.set("spark.sql.legacy.timeParserPolicy","LEGACY")
or
spark.sql("set spark.sql.legacy.timeParserPolicy=LEGACY")
Question: How to draw LineRenderer above other objects?

My line renderer is drawing behind objects. I want it to draw on top of other game objects except for the ball.
How can I do this?
See the following image to reference the problem (the line renderer draws below the goal, and I want it to draw itself on top.

I searched for the issue but haven't found a single answer for 3D.


Answer:
To render a material "above" some other materials, you must set your LineRenderer or TrailRenderer's material Rendering mode to Transparent.
Also, set the Rendering Mode of materials of objects you wish to draw LineRenderer on top to Transparent.
Now go back to the LineRenderer's material and in Advanced Options set its Render Queue to 3999. (higher than the object's materials)
Now your LineRenderer will be drawn on top.
Question: How to get not duplicate rows in join?

I got some problems with duplicate rows which I don't wanna get.
Hi!
I got two tables - tab1, tab2 and I want to join tab2 to tab1 like:
SELECT t1.column_A1, t2.column_B2 
FROM tab1 t1
JOIN
tab2 t2
ON t1.column_A1=t2.column_A2 

tab1
| Column A1 | Column B1 | Column C1 |
|  -------- |  -------- |  -------- |
|  Z1       |  Cell 2   |  Cell 3   |
|  Z2       |  Cell 5   |  Cell 6   |

tab2
| Column A2 | Column B2 | Column C2 |
|  -------- |  -------- |  -------- |
|  Z1       |  PW       |  Cell 3   |
|  Z1       |  RW       |  Cell 6   |

For some rows in tab1 there are more than 1 rows in tab2.
The result will be:
| Column A2 | Column B2 | Column C2 |
|  -------- |  -------- |  -------- |
|  Z1       |  PW       |  RE       |
|  Z1       |  RW       |  KS       |

I want to get:
if PW - show only one row with PW;
if not PW - show only one row with RW
The result should be:
| Column A2 | Column B2 | Column C2 |
|  -------- |  -------- |  -------- |
|  Z1       |  PW       |  RE       |



Answer:
One option is to "sort" rows per each column_a1 by value stored in column_b2 and return rows that rank as the highest.
Sample data:
SQL> WITH
  2     tab1 (column_a1, column_b1, column_c1)
  3     AS
  4        (SELECT 'Z1', 'cell 2', 'cell 3' FROM DUAL
  5         UNION ALL
  6         SELECT 'Z2', 'cell 5', 'cell 6' FROM DUAL),
  7     tab2 (column_a2, column_b2, column_c2)
  8     AS
  9        (SELECT 'Z1', 'PW', 'cell 3' FROM DUAL
 10         UNION ALL
 11         SELECT 'Z1', 'RW', 'cell 6' FROM DUAL
 12         UNION ALL
 13         SELECT 'Z2', 'RW', 'cell 8' FROM DUAL),

Query begins here:
 14     temp
 15     AS
 16        (SELECT t1.column_A1,
 17                t2.column_B2,
 18                ROW_NUMBER () OVER (PARTITION BY t1.column_a1 ORDER BY t2.column_b2) rn
 19           FROM tab1 t1 JOIN tab2 t2 ON t1.column_A1 = t2.column_A2)
 20  SELECT column_a1, column_b2
 21    FROM temp
 22   WHERE rn = 1;

COLUMN_A1    COLUMN_B2
------------ ------------
Z1           PW
Z2           RW

SQL>
Question: Pick error "Type 'Pick' cannot be used as an index type."

Trying to modify my axios wrapper and can't figure out why I'm getting this typescript error...
type Headers = {
  Accept: string;
  'Content-Type': string;
  Authorization?: string;
}

export interface AxiosOptions {
  params?: any;
  data?: any;
  headers: Headers;
}

const axiosOptions: AxiosOptions = {
  headers: {
    Accept: 'application/json',
    'Content-Type': 'application/json',
  },
};

export const updateAxiosOptionsHeaders = (
    key: Pick<Headers, "Accept" | "Content-Type" | "Authorization">,
    value: string
  ): void => {
  axiosOptions.headers[key] = value;
}





Answer:
This doesn't really meet the usage pattern of Pick. You just need keyof Headers for your case:
type Headers = {
  Accept: string;
  'Content-Type': string;
  Authorization?: string;
}

export interface AxiosOptions {
  params?: any;
  data?: any;
  headers: Headers;
}

const axiosOptions: AxiosOptions = {
  headers: {
    Accept: 'application/json',
    'Content-Type': 'application/json',
  },
};

export const updateAxiosOptionsHeaders = (
    key: keyof Headers,
    value: string
  ): void => {
  axiosOptions.headers[key] = value;
}

Playground link

Should you also have different value types in your object (which seems unlikely for a headers object), you can use a generic constraint:
export const updateAxiosOptionsHeaders = <K extends keyof Headers>(
    key: K,
    value: Headers[K]
  ): void => {
  axiosOptions.headers[key] = value;
}