semvec/pypi-packages huggingface dataset:
legacy-taiwan-news-data-tools 0.1.0
Tools for legacy Taiwan news data
License: MIT
Requires Python: >=3.8,<4.0
Keywords: data,decode,encode,language,news,parse

# Tools for Legacy Taiwan News Data

## User guide

### 1. Obtain some legacy news data in Taiwan

- The packed data contains some unreadable characters
- The packed data should be encoded in `big5` or `cp950`
- The packed data extension does not matter
- The packed data should look like this

```plaintext
?坼#ADte   102004/11/21\#AuID    590005\#AuNm    6全玉明\#GrpN    2B1\#ArID    17\#VerN    10\#Hdr1    4test\#Hdr2    0\#Hdr3    0\#ALno    13\#Word    16\#Spec    1N\#ALon    1Y\#Atth    0\#TyID    590005\#TyGR    2B1\#SDte   102004/11/21\#PbID    0?怕K?B
test
　
　◆
?
```

### 2. Install our tools

- Using `pip`

  ```shell
  pip install legacy-taiwan-news-data-tools
  ```

- Using `poetry`

  ```shell
  poetry add legacy-taiwan-news-data-tools
  ```

### 3. Use our tools to process the packed data

- Decode and encode the packed data

  ```python
  from legacy_taiwan_news_data_tools.decode import decode_dict
  from legacy_taiwan_news_data_tools.encode import encode_json
  from io import BytesIO

  packed_data = b'\xc8\xa9\\#ADte   102004/11/21\\#AuID    590005\\#AuNm    6\xa5\xfe\xa5\xc9\xa9\xfa\\#GrpN    2B1\\#ArID    17\\#VerN    10\\#Hdr1    4test\\#Hdr2    0\\#Hdr3    0\\#ALno    13\\#Word    16\\#Spec    1N\\#ALon    1Y\\#Atth    0\\#TyID    590005\\#TyGR    2B1\\#SDte   102004/11/21\\#PbID    0\xc8\xa9\xc8K\xc8B\r\ntest\r\n\xa1@\r\n\xa1@\xa1\xbb\r\n\x00'

  decoded_packed_data = decode_dict(BytesIO(packed_data))

  # Check the decoded data
  assert decoded_packed_data['ADte'] == '2004/11/21'
  assert decoded_packed_data['PbID'] == ''
  assert decoded_packed_data['Data'] == '\u3000\r\n\u3000◆\r\n'

  # Encode the decoded data
  encoded_packed_data = encode_json(decoded_packed_data)

  # Check the encoded data
  assert encoded_packed_data == b'{"ADte":"2004/11/21","AuID":"90005","AuNm":"\xe5\x85\xa8\xe7\x8e\x89\xe6\x98\x8e","GrpN":"B1","ArID":"7","VerN":"0","Hdr1":"test","Hdr2":"","Hdr3":"","ALno":"3","Word":"6","Spec":"N","ALon":"Y","Atth":"","TyID":"90005","TyGR":"B1","SDte":"2004/11/21","PbID":"","Data":"test\\r\\n\xe3\x80\x80\\r\\n\xe3\x80\x80\xe2\x97\x86\\r\\n"}'
  ```