Repository: microsoft/semantic-kernel
File: dotnet/src/SemanticKernel.Core/Text/TextChunker.cs (C#)

Code before:

        public void AddRange(StringListWithTokenCount range) => this.Values.AddRange(range.Values);

        public void RemoveRange(int index, int count) => this.Values.RemoveRange(index, count);

        public int Count => this.Values.Count;

        public List<string> ToStringList() => this.Values.Select(v => v.Value).ToList();

        private List<(string Value, int TokenCount)> Values { get; } = [];

        public string ValueAt(int i) => this.Values[i].Value;

        public int TokenCountAt(int i) => this.Values[i].TokenCount;
    }

    /// <summary>
    /// Delegate for counting tokens in a string.
    /// </summary>
    /// <param name="input">The input string to count tokens in.</param>
    /// <returns>The number of tokens in the input string.</returns>
    public delegate int TokenCounter(string input);

    private static readonly char[] s_spaceChar = [' '];
    private static readonly string?[] s_plaintextSplitOptions = ["\n\r", ".。．", "?!", ";", ":", ",，、", ")]}", " ", "-", null];
    private static readonly string?[] s_plaintextLineSplitOptions = ["\n", ".。．", "?!", ";", ":", ",，、", ")]}", " ", "-", null];
    private static readonly string?[] s_markdownSplitOptions = [".\u3002\uFF0E", "?!", ";", ":", ",\uFF0C\u3001", ")]}", " ", "-", "\n\r", null];

    /// <summary>
    /// Split plain text into lines.
    /// </summary>
    /// <param name="text">Text to split</param>
    /// <param name="maxTokensPerLine">Maximum number of tokens per line.</param>
    /// <param name="tokenCounter">Function to count tokens in a string. If not supplied, the default counter will be used.</param>
    /// <returns>List of lines.</returns>
    public static List<string> SplitPlainTextLines(string text, int maxTokensPerLine, TokenCounter? tokenCounter = null)
    {
        // First, always split on newlines regardless of token count
        text = text.Replace("\r\n", "\n"); // normalize line endings
        var lines = text.Split('\n');

        // If we only have one line, use the existing logic with line-specific split options
        if (lines.Length == 1)
        {
            return InternalSplitLines(text, maxTokensPerLine, trim: true, s_plaintextLineSplitOptions, tokenCounter);
        }

        // Apply token-based splitting to each line if needed
        var result = new List<string>();
        foreach (var line in lines)
        {

Review comment (bug):
This is not actually a bug, I though it was too at first glance, but if we follow the code down to when each entry get's used, it's used as a Span<char> and will in fact split on either `\n` or `\r`. It's just confusing since they are written in the reverse of how we normally see them, and the input being strings isn't usually how we see an arrya of chars getting defined (at least IMO). But we don't need to define a second version of this. Since the text is also already being normalized, we could simplify the existing definition and remove the `\r` from the string.

Code after:

        public void AddRange(StringListWithTokenCount range) => this.Values.AddRange(range.Values);

        public void RemoveRange(int index, int count) => this.Values.RemoveRange(index, count);

        public int Count => this.Values.Count;

        public List<string> ToStringList() => this.Values.Select(v => v.Value).ToList();

        private List<(string Value, int TokenCount)> Values { get; } = [];

        public string ValueAt(int i) => this.Values[i].Value;

        public int TokenCountAt(int i) => this.Values[i].TokenCount;
    }

    /// <summary>
    /// Delegate for counting tokens in a string.
    /// </summary>
    /// <param name="input">The input string to count tokens in.</param>
    /// <returns>The number of tokens in the input string.</returns>
    public delegate int TokenCounter(string input);

    private static readonly char[] s_spaceChar = [' '];
    private static readonly string?[] s_plaintextSplitOptions = ["\n", ".。．", "?!", ";", ":", ",，、", ")]}", " ", "-", null];
    private static readonly string?[] s_markdownSplitOptions = [".\u3002\uFF0E", "?!", ";", ":", ",\uFF0C\u3001", ")]}", " ", "-", "\n\r", null];

    /// <summary>
    /// Split plain text into lines.
    /// </summary>
    /// <param name="text">Text to split</param>
    /// <param name="maxTokensPerLine">Maximum number of tokens per line.</param>
    /// <param name="tokenCounter">Function to count tokens in a string. If not supplied, the default counter will be used.</param>
    /// <returns>List of lines.</returns>
    public static List<string> SplitPlainTextLines(string text, int maxTokensPerLine, TokenCounter? tokenCounter = null) =>
        InternalSplitLines(text, maxTokensPerLine, trim: true, s_plaintextSplitOptions, tokenCounter);

    /// <summary>
    /// Split markdown text into lines.
    /// </summary>
    /// <param name="text">Text to split</param>
    /// <param name="maxTokensPerLine">Maximum number of tokens per line.</param>
    /// <param name="tokenCounter">Function to count tokens in a string. If not supplied, the default counter will be used.</param>
    /// <returns>List of lines.</returns>
    public static List<string> SplitMarkDownLines(string text, int maxTokensPerLine, TokenCounter? tokenCounter = null) =>
        InternalSplitLines(text, maxTokensPerLine, trim: true, s_markdownSplitOptions, tokenCounter);

    /// <summary>
    /// Split plain text into paragraphs.
    /// </summary>
    /// <param name="lines">Lines of text.</param>