This comment form is no longer interactive because the comment period is closed.

2016-01 Modifications to TOP and IRO Standards | IRO-002-5 and TOP-001-4

Description:

Start Date: 08/31/2016
End Date: 10/17/2016

Associated Ballots:

Ballot Name Project Standard Pool Open Pool Close Voting Start Voting End
2016-01 Modifications to TOP and IRO Standards TOP-001-4 AB 2 ST 2016-01 Modifications to TOP and IRO Standards TOP-001-4 06/20/2016 07/19/2016 10/05/2016 10/17/2016
2016-01 Modifications to TOP and IRO Standards IRO-002-5 AB 2 ST 2016-01 Modifications to TOP and IRO Standards IRO-002-5 06/20/2016 07/19/2016 10/05/2016 10/17/2016
2016-01 Modifications to TOP and IRO Standards IRO-002-5 Non-binding Poll AB 2 NB 2016-01 Modifications to TOP and IRO Standards IRO-002-5 Non-binding Poll 06/20/2016 07/19/2016 10/05/2016 10/17/2016
2016-01 Modifications to TOP and IRO Standards TOP-001-4 Non-binding Poll AB 2 NB 2016-01 Modifications to TOP and IRO Standards TOP-001-4 Non-binding Poll 06/20/2016 07/19/2016 10/05/2016 10/17/2016

Filter:

Hot Answers

Thomas Lyons, Owensboro Municipal Utilities, 3, 10/17/2016

- 0 - 0

Daniel Herring, DTE Energy - Detroit Edison Company, 4, 10/17/2016

- 0 - 0

Other Answers

Michael Puscas, ISO New England, Inc., 2, 9/21/2016

- 0 - 0

- 0 - 0

Thomas Foltz, AEP, 5, 10/4/2016

- 0 - 0

Diana McMahon, On Behalf of: Salt River Project, WECC, Segments 1, 3, 5, 6

- 1 - 0

While Clark is not an RC, it believes its arguments expressed in question 2 below are also applicable to the RC and that any similar requirements and measures applicable to the RC in IRO-002-5 should be similarly modified.

Jack Stamper, Clark Public Utilities, 3, 10/5/2016

- 0 - 0

We agree with the proposed changes but have a suggestion for a minor revision to the language of R1.

It’s very hard to enforce a standard that requires the Entity to “have” data exchange capability. While we don’t need to require a formal procedure document, it should be clear that to comply with the standard, the Entity will be required to provide documented evidence as set out in M1.  This comment also applies to R2.

Suggested change: delete the word have and replace with document and implement

R1. Each Reliability Coordinator shall have document and implement data exchange capabilities with its Balancing Authorities and Transmission Operators, and with other entities it deems necessary, for it to perform its Operational Planning Analyses. [Violation Risk Factor: Medium] [Time Horizon: Operations Planning].

Steven Rueckert, Western Electricity Coordinating Council, 10, 10/7/2016

- 0 - 0

Richard Vine, 10/10/2016

- 0 - 0

We believe that Requirement R5 should identify how these non-BES facilities are determined, such as through Seasonal Assessments and other Monthly Analysis.  In our opinion, in no case should this be left open ended, without bounds.

David Jendras, Ameren - Ameren Services, 3, 10/11/2016

- 0 - 0

- 0 - 0

Karie Barczak, On Behalf of: DTE Energy - Detroit Edison Company, , Segments 3, 4, 5

- 0 - 0

GSOC supports Southern Company's comments. 

Questions

Do you agree with the changes made by the SDT to draft standard IRO-002-5? If you do not agree, or if you agree but have comments or suggestions for the proposed standard provide your recommendation and explanation.

 

 No

 

Provide any additional comments for the SDT to consider, if desired.

 

Comments:   Southern believes that the language in IRO-002-5 R2 explicitly limits the scope of the requirement to “the data exchange infrastructure inside the Primary Control Center”. The first problem here is that the term “data exchange infrastructure” has no clear or broadly accepted industry definition.  

The second problem is that here is no clear definition of what constitutes the control center. Is it a facility or a room inside a facility? What prevents someone from moving the “capability” outside the control center (i.e a data center not part of the control center)?

 

The language in IRO-002-5 R3 currently has a requirement to test the “primary control center data exchange capabilities” specified in R2” every 90 days. First of all, the terminology shifts from the word “infrastructure” in R2 to “capabilities” in R3, which leaves a lot of ambiguity. Why establish a requirement to test the redundancy of the data exchange and not the EMS platform in which the capability resides.  This is perplexing given the fact that the data exchange function is, in most cases, a sub-component of the much larger distributed EMS architecture.

 

The language in IRO-002-5 R6 is also confusing,  as it states that  “each RC shall have monitoring systems that provide information utilized by the RC’s operating personnel, giving particular emphasis to alarm management and awareness systems, automated data transfers, and synchronized information systems, over a redundant infrastructure.  Again, this is very confusing, because the following questions have not been answered:

 

  • What is meant by “particular emphasis”?
  • Which “awareness systems” require redundant infrastructure
  • Which “automated data transfers” are in scope?
  • Which “synchronized information”  is in scope?
  • What level of redundancy is required? Server level, component level, network level, etc.

 

 

Scott McGough, Georgia System Operations Corporation, 3, 10/11/2016

IRO-002-5 SOCO Comments.docx

- 0 - 0

Xcel Energy believes that there is still some clarification needed to the definition of data exchange.  Is it meant to cover data exchange between control centers (ICCP) or does this include RTUs and Communication paths?

Amy Casuscelli, On Behalf of: Xcel Energy, Inc. - MRO, WECC, SPP RE - Segments 1, 3, 5, 6

- 0 - 0

Michelle Amarantos, APS - Arizona Public Service Co., 5, 10/12/2016

- 0 - 0

Comments: See comments below.

MEAG Power voted Affirmative in error and requests that its Affirmative vote be changed to Negative for all associated ballots, Standard changes and Non-Binding opinions.  MEAG Power adopts and supports the comments of Southern Company.

Regards,

Scott Miller, Proxy, MEAG Power, 678-644-3524

- 0 - 0

Colorado Springs Utilities, Segment(s) 1, 6, 3, 5, 9/14/2015

- 0 - 0

Joel Wise, On Behalf of: Tennessee Valley Authority - SERC - Segments 1, 3, 5, 6

- 0 - 0

- 0 - 0

R4:  Is the intent of the requirement to give System Operators the authority to deny planned outages and maintenance of telecommunication, monitoring and analysis capabilities, in the Real-Time Operations and Same-Day Horizons?  It is hard to see the benefit of having shift System Operators in on the approval process of planned work of this type versus dedicated support staff that can evaluate this type of work and approve or deny the work during the Operations Planning Time Horizon;  must System Operators be involved in the approval of this type of work in the Operations Planning Time Horizon?  The requirement is difficult to understand since ‘approve’ is used instead of ‘deny,’ and three Time Horizons are listed as applicable.

R6:  The phrase ‘giving particular emphasis to alarm management and awareness systems…’ is vague, ambiguous, and un-measurable, and makes interpretation of the standard difficult.  This type of language has historically been eliminated from several standards under the Paragraph 81 criteria.

Are redundant functionality mentioned in R3 and redundant infrastructure mentioned in R6 two different things?  Neither are defined terms and make interpretation of this standard more difficult.

Duke Energy , Segment(s) 1, 5, 6, 4/10/2014

- 0 - 0

Andrew Pusztai, 10/14/2016

- 0 - 0

Southern believes that limiting the scope of Requirement R2 to “the data exchange infrastructure inside the Primary Control Center” may allow for entities to circumvent the requirements by moving their data exchange infrastructure to a physical location outside of their control center (i.e., a remote data center).  It is important for the SDT to ensure the reliability intent of Requirement R2 is maintained by focusing on the “data exchange capability” for the primary control center regardless of where the actual data exchange infrastructure physically resides.  To remove any ambiguity, it is recommended that the SDT define the following terms:

 

  • Data Exchange Infrastructure

  • Data Exchange Capability

  • Primary Control Center

 

In regard to R3, it is Southern Company's understanding that in most cases (including ours) that the data exchange infrastructure is an integrated component of the EMS infrastructure. Southern Company does not understand the purpose of having a requirement for redundant infrastructure for data exchange but not for the EMS.

 

Southern Company also requests clarification regarding the language used in IRO-002-5 R6, which states  “each RC shall have monitoring systems that provide information utilized by the RC’s operating personnel, giving particular emphasis to alarm management and awareness systems, automated data transfers, and synchronized information systems, over a redundant infrastructure.”  Some questions regarding the proposed requirement are as follows:

 

  • What is meant by “particular emphasis”?

  • Which “awareness systems” require redundant infrastructure?

  • Which “automated data transfers” are in scope?

  • Which “synchronized information”  is in scope?

  • What level of redundancy is required? Server level, component level, network level, etc.

 

Southern Company, Segment(s) 1, 3, 5, 6, 8/3/2016

- 0 - 0

The comments provided by the NSRF that are applicable to TOP-001-4 are also applicable to IRO-002-5 for similar requirements. 

MRO-NERC Standards Review Forum (NSRF), Segment(s) 3, 4, 5, 6, 1, 2, 7/14/2016

- 0 - 0

The requirements should apply to “secondary” Control Centers as well.  There are times when a primary Control Center might be out of service for a prolonged length of time, and the “secondary” Control Center must have the capabilities addressed by this standard.  If there is another standard that addresses “secondary” Control Centers, the Purpose of IRO-002-5 should reflect that it only applies to a primary Control Center.  If IRO-002-5 is left with “primary”, then primary Control Center will need to be defined in the NERC Glossary.

RSC no National Grid, Segment(s) 1, 0, 2, 4, 5, 6, 7, 3, 10/14/2016

- 0 - 0

Karl Blaszkowski, CMS Energy - Consumers Energy Company, 3, 10/14/2016

- 0 - 0

Karl Blaszkowski, CMS Energy - Consumers Energy Company, 3, 10/14/2016

- 0 - 0

Karl Blaszkowski, CMS Energy - Consumers Energy Company, 3, 10/14/2016

- 0 - 0

Karl Blaszkowski, CMS Energy - Consumers Energy Company, 3, 10/14/2016

- 0 - 0

<p>Texas RE appreciates the Standard Drafting Team&rsquo;s efforts to develop a workable approach to requiring redundant and diversely routed data exchange infrastructure.&nbsp; However, Texas RE is concerned that the SDT&rsquo;s proposed approach limiting such diverse routing requirements solely to primary control centers is overly narrow.&nbsp; Texas RE requests that the SDT apply the diverse routing requirements at issue here to control centers generally, rather than to just the primary control center.&nbsp; However, if the SDT declines to do so, Texas RE requests that the SDT clarify the relationship between TOP-001-4 and IRO-002-5 with the backup functionality requirements set forth in EOP-008.&nbsp;</p><p>In Order No. 693, FERC made clear that entities should possess backup capabilities that, among other things, &ldquo;provide for a minimum set of tools and facilities to replicate the critical reliability functions of the primary control center.&rdquo;&nbsp; (p. 160, &para;335).&nbsp; In Order No. 817, FERC further identified a clear &ldquo;reliability need for the reliability coordinator, transmission operator, and balancing authority to have data exchange capabilities that are redundant and diversely routed.&rdquo;&nbsp; (p. 33, &para;47).&nbsp; Given the clear directive that entities possess backup control centers that can replicate the reliability functions of the primary control center, it seems contrary to the general ERO-wide approach to backup functionality to only require diverse routing within primary control centers.</p><p>This also appears counter to FERC&rsquo;s specific discussion of the relationship between the general backup functionality requirements in EOP-008 and the more specific requirements for voice communications in the COM Standards and the data exchange capability standards at issue here.&nbsp; Specifically, in Order No. 817, FERC made clear that the EOP-008 redundancy requirements should not supplant the diverse routing obligations to be set forth in the revised TOP and IRO compliance obligations.&nbsp; That is to say, although it is possible to read the EOP-008 backup functionality requirements as mandating sufficient redundancy in and of itself, FERC nevertheless called for the diverse routing reliability need to be explicitly addressed in the TOP/IRO Standards in the same manner as voice communications were addressed under the COM Standards.&nbsp; However, FERC&rsquo;s directive does not appear to contemplate simply eliminating the diverse routing requirements from the TOP/IRO Standards (and arguably to EOP-008 Standard as well) altogether.</p>

Rachel Coyne, Texas Reliability Entity, Inc., 10, 10/14/2016

- 0 - 0

1.       Abstain (standard is not applicable to HONI)

- 0 - 0

See comments to question #2

- 0 - 0

What does the SDT consider “data exchange infrastructure”?  Without an understanding of the intent of this language, it is unclear where the expectation for “redundant and diversely routed” ends.  If the intent is to require the same level of demonstration of evidence as was provided under the old COM-001-1, then redundancy typically only had to be demonstrated by showing the two separate telecomm lines going to two separate routers and then from there it went into the single firewall and then into the ESP.  If the ‘primary Control Center’ is considered within the single firewall/ESP boundary, then that should be clarified further in the requirement. 

Suggested changes to R2:

R20. Each Transmission Operator shall have data exchange capabilities, with redundant and diversely routed data exchange infrastructure after the point the data enters the Transmission Operator's primary Control Center, for the exchange of Real-time data with its Reliability Coordinator, Balancing Authority, and the entities it has identified it needs data from in order for it to perform its Real-time monitoring and Real-time Assessments.

We also would like to see the 90 day requirement to test changed to ‘Quarterly’. 

We have potential concerns (related to ‘where’ the boundary is considered for the Control Center) about which components need to be tested and what is considered an adequate test.  Without knowing what components are included we may not test the right things.

SPP Standards Review Group, Segment(s) , 10/14/2016

- 0 - 0

FMPA, Segment(s) , 8/3/2016

- 0 - 0

Scott Downey, 10/14/2016

- 0 - 0

Jerome Gobby, 10/14/2016

- 0 - 0

(1)   We thank the SDT for responding to our request to clarify that testing of data exchange capabilities should only apply to the primary Control Center and at a required frequency greater than monthly.

(2)   We believe the proposed requirements should follow a more performance-based approach and utilize the associated VSLs to identify the severity of non-compliance.  In its current form, a registered entity could instantly become non-compliant if these data exchange capabilities or associated analytical tools become unavailable.  We recommend that R1 be reworded to state “Each RC shall maintain data exchange capabilities with its BAs, its TOPs, and other entities it deems necessary, to perform its Operational Planning Analyses.”

(3)   Likewise, we recommend that R2 be reworded to state “Each RC shall maintain data exchange capabilities, with redundant and diversely routed data exchange infrastructure within the RC’s primary Control Center, for the exchange of Real-time data with its BAs, its TOPs, and with other entities it deems necessary, to perform its Real-time monitoring and Real-time Assessments.”

(4)   We believe compliance should be embedded within existing business processes to better adopt such practices within a registered entity’s operations.  Many registered entities already execute or follow the execution of quarterly processes, and we believe the testing of data exchange capabilities could be included in such processes like quarterly model updates.  The tracking of every 90 days could be cumbersome for registered entities to coordinate test schedules and staffing levels for adequate test participation in advance.  Moreover, it may be possible that two tests are conducted within the same quarter, something the SDT is likely trying to avoid, and could fall during operating periods that are of high risk to the BES.   We recommend the periodicity of these tests, as identified in R3, be changed to calendar quarters.

ACES Standards Collaborators, Segment(s) 1, 3, 5, 6, 4, 10/14/2016

- 0 - 0

sean erickson, Western Area Power Administration, 1, 10/14/2016

- 0 - 0

Please see comments in response to Question #5.

Elizabeth Axson, 10/14/2016

- 0 - 0

Cain Braveheart, On Behalf of: Bonneville Power Administration, WECC, Segments 1, 3, 5, 6

- 0 - 0

Douglas Webb, 10/14/2016

- 0 - 0

Hot Answers

. Documented assessments every 30 minutes is an unnecessary administrative burden.

Thomas Lyons, Owensboro Municipal Utilities, 3, 10/17/2016

- 0 - 0

Daniel Herring, DTE Energy - Detroit Edison Company, 4, 10/17/2016

- 0 - 0

Other Answers

Michael Puscas, ISO New England, Inc., 2, 9/21/2016

- 0 - 0

 

We appreciate the work that the Drafting Team has provided specific to the TOP-001-4 Reliability Standard.  However, our concern remains over the test language found in Requirements R21 for the TOP and R24 for the BA.  While we understand that monitoring functions do not constitute sufficiency in testing we are concerned that the “test” terminology is subject to interpretation by the responsible entity, auditor, and others that lend itself to inconsistent implementation/auditing of these requirements.  To resolve this reliability concern, please clarify the drafting team’s intent  for the “test” requirement whether this is explicit to each data link or the data link infrastructure or some other intention.  We, Sacramento Municipal Utility District and Balancing Authority Northern California, look forward to a resolution from the drafting team on this issue…  Thanks.

 

 

 

- 0 - 0

AEP recognizes FERC’s concerns regarding identification of non-BES facilities, however, there would be far more flux involved in their identification and real-time monitoring (as suggested by the SAR) than may be widely understood or appreciated. This subset of non-BES facilities would change quite frequently, and creating obligations to govern such frequently changing identification and real-time monitoring would likely require much effort, with little to no improvement in reliability. The Time Horizon for R10 is “Real-Time Operations”, and while the monitoring of non-BES facilities may be accomplished in Real-Time, their identification cannot be. Some sort of methodology or guidance should be provided for the monitoring of non-BES facilities and the associated data,  specifically data from outside the Transmission Operator’s area. As previously stated, rather than developing additional requirements which would not likely be beneficial, we continue to believe a more prudent approach is to focus on the desired end state itself. We still believe the argument can still be made that our existing obligations, when considered as a whole, could collectively appease FERC’s concerns. 

While we appreciate the SDT’s recent revisions which no longer requires monthly testing, we once again recommend using the text “once a calendar quarter” rather than “every 90 calendar days” as is most recently proposed.

Thomas Foltz, AEP, 5, 10/4/2016

- 0 - 0

Requirement 20 defines the needs for "data exchange infrastructure". SRP feels that this is prescribing a technology solution for the need for redundant and diversly routed data exchange capabilites that is implied. Entities must be responsbile for determining the method for obtaining compliance. The standards need to refrain from defining a solution that may be difficult to obtain for all entities.

Diana McMahon, On Behalf of: Salt River Project, WECC, Segments 1, 3, 5, 6

- 0 - 0

Clark believes that R20 is still not addressing the issue FERC has expressed its concern over. Please read the paragraph in question. In Paragraph 47, Order 817, FERC states:

“We agree with NERC and other commenters that there is a reliability need for the reliability coordinator, transmission operator and balancing authority to have data exchange capabilities that are redundant and diversely routed. However, we are concerned that the TOP and IRO Standards do not clearly address redundancy and diverse routing so that registered entities will unambiguously recognize that they have an obligation to address redundancy and diverse routing as part of their TOP and IRO compliance obligations. NERC’s comprehensive approach to establishing communications capabilities necessary to maintain reliability in the COM standards is applicable to data exchange capabilities at issue here. Therefore, pursuant to section 215(d)(5) of the FPA, we direct NERC to modify Reliability Standards TOP-001- 3, Requirements R19 and R20 to include the requirement that the data exchange capabilities of the transmission operators and balancing authorities require redundancy and diverse routing. In addition, we direct NERC to clarify that “redundant infrastructure” for system monitoring in Reliability Standards IRO-002-4, Requirement R4 is equivalent to redundant and diversely routed data exchange capabilities.”

The SDT continues to INCORRECTLY apply this paragraph first to control centers and now to primary control centers. Paragraph 47 is applicable to the registered the entities RCs, TOPs, and BAs and requires these registered entities to have the referenced redundancy and diverse routing of data exchange. The terms “control center” or "primary control center" are not used in Paragraph 47. The SDT continues to fail to recognize that many RCs, TOPs, and BAs already have redundancy and diverse routing of data exchange that addresses the FERC's concerns because they have voluntarily adopted this approach in meeting their COM standards compliance and their EOP-008 compliance.

In Paragraph 48, Order 817, FERC states:

“Further, we disagree with commenter arguments that Reliability Standard EOP-008-1 provides alternatives to data exchange redundancy and diverse routing. The NERC standard drafting team that developed the COM standards addressed this issue in the standards development process, responding to a commenter seeking clarification on the relationship between communication capabilities, alternative communication capabilities, primary control center functionality and backup control center functionality. The standard drafting team responded that “Interpersonal Communication and Alternative Interpersonal Communication are not related to EOP-008,” even though Reliability Standard EOP-008-1 Requirement R1 applies equally to data communications and voice communications. To the extent the standard drafting team asserted that Reliability Standard EOP-008 did not supplant the redundancy requirements of the COM Reliability Standards, we believe the same is true for data communications. Redundancy for data communications is no less important than the redundancy explicitly required in the COM standards for voice communications.”

In Paragraph 48 the FERC DID NOT state that the use of a backup dispatch center in the provision of alternatives to data exchange redundancy and diverse routing fails to address its concerns expressed in Paragraph 47 . It only stated that the requirement to have such data exchange redundancy and diverse routing is not specifically provided for in EOP-008-1. There is no reason for the SDT to believe that the requirements of Paragraph 47 should be applied to the primary control center. There is every reason to believe that the requirements of Paragraph 47 should be applied to the RCs, TOPs, and BAs provision of reliability data using data exchange capabilities that are redundant and diversely routed.

As such the SDT needs to modify R20 and M20 to the following:

“R20. Each Transmission Operator shall have data exchange capabilities, with redundant and diversely routed data exchange infrastructure for the exchange of Real-time data with its Reliability Coordinator, Balancing Authority, and the entities it has identified it needs data from in order for it to perform its Real-time monitoring and Real-time Assessments. [Violation Risk Factor: High] [Time Horizon: Same-Day Operations, Real-time Operations]”

“M20. Each Transmission Operator shall have, and provide upon request, evidence that could include, but is not limited to, system specifications, system diagrams, or other documentation that lists its data exchange capabilities, including redundant and diversely routed data exchange infrastructure for the exchange of Real-time data with its Reliability Coordinator, Balancing Authority, and the entities it has identified it needs data from in order to perform its Real-time monitoring and Real-time Assessments as specified in the requirement.”

The above revisions to R20 and M20 will cause Transmission Operators to “unambiguously recognize that they have an obligation to address redundancy and diverse routing as part of their TOP and IRO compliance obligations” which is the concern FERC has expressed. These changes will allow TOPs to submit evidence that the redundant and diversely routed data exchange capabilities being voluntarily used in compliance with the COM standards and EOP-008-1 are also capable of meeting the new MANDATORY requirements of TOP-001-4. The SDT should have no concern that the FERC would reject this since Paragraph 47 clearly DOES NOT REQUIRE THIS REDUNDANCY AND DIVERSE ROUTING OF DATA EXCHANGE FOR PRIMARY CONTROL CENTERS BUT ONLY FOR RCs, TOPs, and BAs as registered entities.

While Clark is not an RC or a BA, it believes the above arguments are also applicable to these registered entities and that any similar requirements and measures applicable to these entities in IRO-002-5 or TOP-001-4 should be similarly modified.

Jack Stamper, Clark Public Utilities, 3, 10/5/2016

- 0 - 0

Steven Rueckert, Western Electricity Coordinating Council, 10, 10/7/2016

- 0 - 0

 

New requirement R20 requires TOPs to have “redundant and diversely routed data exchange infrastructure within the Transmission Operator’s primary Control Center.”   R23 contains a similar requirement for BAs to have similar infrastructure “within the Balancing Authority’s primary Control Center.  It has been the ISO’s understanding that the concern was with redundancy into and out of the Control Center, to and from the outside world.   However, the terminology “within the control center” could be construed differently that there is some expectation of diversity and redundancy before the data leaves the control center.   The ISO requests that the Drafting Team clarify whether or not the intent of this terminology is to apply to routing after the data leaves the Control Center.  If indeed the new requirement is meant to apply within the Control Center, the ISO requests that more specificity be provided as to what the expectation is for redundancy and diversity (i.e. -  Between one operator and another? Between an operator computer and or phone, and the data exchange infrastructure itself? )

Richard Vine, 10/10/2016

- 0 - 0

For Requirement  R10.6 we are concerned how non-BES facilities outside the Transmission Operator Area should be identified by the TOP?  We believe that this should be specified.  This is partly mentioned in the Rationale for Requirement 10; but is not part of this requirement.  The requirement should be set up to complement the other standards’ requirements.

David Jendras, Ameren - Ameren Services, 3, 10/11/2016

- 0 - 0

Oncor’s comments have no change from the last draft as there is no change to the Standard Requirement in this draft.  Revisions made in rationale boxes do not change the requirement and often are removed when the Standard becomes effective.

Proposed TOP-001-4 R10 requires TOP’s to monitor its facilities, Remedial Action Schemes and Non-BES facilities that it identifies as necessary to determine SOL exceedances in R10.1, R10.2 and R10.3.  For Sub-Requirements R10.4, R10.5 and R10.6 the wording has changed to “obtain and utilize” instead of the former “monitor” used in previous drafts of TOP-001-3. These Sub-Requirements also use the wording “identified as necessary by the Transmission Operator”.  The proposed TOP-001-4 RSAW requires the Transmission Operator to provide evidence that it monitored all the data stated in the Sub-Requirements without requiring the TOP to providing reasoning or qualifications for how the TOP determined what or how the data “obtained and utilized” was “identified as necessary”.  This creates unenforceable requirements that have no reason to be added to a Standard.

Proposed TOP-001-4 R10.5 requires TOPs to obtain and utilize statuses of Remedial Action Schemes in neighboring TOP areas.  Currently TOP SPS statuses is communicated through notifications required to the RC and affected TOPs.  This notification process requirement works and keeps the wide area system monitoring and control responsibility on ERCOT the Reliability Coordinator and not on individual TOPs.

In closing, the ERCOT region is structured to support a deregulated market in which ERCOT monitors facilities for all TOPS and has a centralized view of the entire region to maintain reliability. TOPs operating within ERCOT currently do not have the technical capability to obtain and utilize data specified in R10.4, R10.5 and R10.6. This requirement imposes a "one size fits all" regional structure which would place an unreasonable financial burden on all TOPs to both install and maintain additional hardware in each station or install and maintain multiple ICCPs between control centers. This requirement would place this financial burden on TOPs for nothing more than to replicate an RC function with no benefit to the BES. At no point in proposed Standard TOP-001- 4 does it require TOs to supply neighboring TOs with this data. Oncor requests R1O.4, R10.5, R10.6 be removed from the standard due to lack of regional flexibility.

 

- 0 - 0

Karie Barczak, On Behalf of: DTE Energy - Detroit Edison Company, , Segments 3, 4, 5

- 0 - 0

R20 Each Transmission Operator shall have data exchange capabilities, with redundant and diversely routed data exchange infrastructure within the Transmission Operator's primary Control Center, for the exchange of Real-time data with its Reliability Coordinator, Balancing Authority, and the entities it has identified it needs data from in order for it to perform its Real-time monitoring and Real-time Assessments.

 

{C}1.      {C}This requires redundancy and diverse routing only within the Transmission Operator’s primary control center.  The communication links referenced could easily cover hundreds of miles, and they can only be well protected during the last few hundred feet within an entities facilities.  This requirement says you must have redundancy and diversity for the short and well controlled portion, but not for the much longer, less controlled, and thus more vulnerable portion.  I cannot see a technical basis for this approach.

{C}2.      {C}The term “diversely routed” as applied to the area within a Control Center is not well defined.  For example, can the two cables be within the same cable tray for a short distance?  If so, for how long?

{C}3.      {C}The extent of redundancy required is not clear.  Would this require redundancy in the servers that are the source of the data, would it require redundant Ethernet ports on such a server, would it require redundancy in the power supplies supporting the equipment?  All these questions (and more) must be answered for an entity to design a compliant solution.   

{C}4.      {C}Has any work been done to determine how frequently a failure within a control center is the cause of a data communications failure?  It would seem necessary to do something like that before implementing this requirement.

{C}5.      {C}The rationale implies that failure to have redundant facilities in place when one set of facilities is being upgraded is not a violation of the requirement, but there is nothing in the actual requirement to support that.  That is not an appropriate use of the rationale section.  Consider adding the following at the beginning of the requirement: “Except during planned outages of less than two weeks duration and during unplanned outages, . . . ”

{C}6.      {C}This requirement is focused on infrastructure.  It is not clear how the availability of a completely different approach to the data exchange might meet this requirement.  If the data set was small enough that it could be verbally exchanged via telephone, faxed, emailed, or FTP’d to the other party, would that meet the requirement?  Consider rewording the requirement to state the objective more broadly and therefore allow these approaches to be used to meet compliance.

{C}7.      {C}The requirement does not take into account variability in the criticality of the data.  Where failure of a data link to one entity might be a minor annoyance, failure of a link to another might be catastrophic.  This requirement allows no flexibility in matching the degree of redundancy and diversity to the associated risk of the loss of the data.  Consider giving the TOP latitude to determine the appropriate level of redundancy for a particular link, particularly in consideration of the criticality of the data and the reliability of various parts of the data exchange infrastructure.

 

Rationale for Requirement R21: The proposed requirement addresses directives for testing of data exchange capabilities used in primary Control Centers (FERC Order No. 817 Para 51).

A test for redundant functionality demonstrates that data exchange capabilities will continue to operate despite the malfunction or failure of an individual component. An entity's testing practices should, over time, examine the various failure modes of its data exchange capabilities. When an actual event successfully exercises the redundant functionality, it can be considered a test for the purposes of the proposed requirement.

 

1.     The second sentence in the second paragraph should not be part of the rationale of the standard.  The rationale should include the reasoning behind the standard.  This statement attempts to add an element to the requirement and thus should be part of the requirement.  As it stands now it will introduce confusion over whether incorporating various failure modes is mandatory or advisory.

2.     Similarly the third sentence in the second paragraph should not be part of the rationale of the standard.  If an actual event can be substituted for a test, that should be made part of the requirement as it is done in other standards.  See CIP-009, R2 for an example of how this can be accomplished correctly.

 

 

Scott McGough, Georgia System Operations Corporation, 3, 10/11/2016

- 0 - 0

Amy Casuscelli, On Behalf of: Xcel Energy, Inc. - MRO, WECC, SPP RE - Segments 1, 3, 5, 6

- 0 - 0

R10. In the previous version of TOP-001-4, R10 required TOPs to monitor non-BES facilities necessary for determining SOL exceedances both within and outside its footprint “identified as necessary by the TOP.” This has been broadened to include the RC with the introduction of new text in the “Rationale for Requirement R10” box; which states that any of the following could lead to an identification of non-BES facilities that should be monitored:

  1. APS OPA
  2. APS RTA
  3. APS analysis to determine BES Inclusion Exceptions
  4. Peak RC analysis performed in support of outage coordination (requiring temporary monitoring) 

This is problematic for several reasons:

  1. The RC model may not be as robust as the TOP’s model for non-BES facilities. Therefore, the RC’s ability to determine which non-BES facilities affect the BES is limited without TOP input. A "vetting" process should be added whereby the RC must validate the non-BES facilities to be added with the TOP.
  2. Temporary monitoring of non-BES facilities, if the IT infrastructure is not already in place, may not be feasible depending on the lead time available in the RC’s outage coordination process; particularly if the facilities are not already modeled and/or the data is not readily available. For example, in the Western Interconnection, Peak Reliability initiates their OPA studies only two days before the scheduled outage. 

At a minimum, if support of the RC outage coordination process is to remain as part of requirement R10, then the text of the requirement should be expanded to specifically include the RC: “identified as necessary by the TOP or by the RC in support of their outage coordination process.” 

R20 and R23. The introduction of the word “primary” as a clarifier to Control Center is problematic in that it limits where the an entity may locate its “redundant and diversely routed data exchange infrastructure” to that within the “primary Control Center.” As APS has redundant and diversely routed data exchange infrastructure across (and not within) Control Centers; i.e. infrastructure that spans our primary and back-up Control Center locations, the requirement as written limits flexibility in terms of where redundant infrastructure may be located. As worded, this would require entities to install additional redundancy within its primary Control Center location.

If the SDT’s intent is to ensure the reliability of data exchange structure used to maintain its data exchange operations within the primary control Center, we propose modifying the language as follows to recognize redundant data exchange capability infrastructure across an entity’s collective Control Center facilities: 

APS Proposed R20. Each Transmission Operator shall have data exchange capabilities, with redundant and diversely routed data exchange infrastructure used to maintain its operations within the Transmission Operator's primary Control Center, for the exchange of Real-time data with its Reliability Coordinator, Balancing Authority, and the entities it has identified it needs data from in order for it to perform its Real-time monitoring and Real-time Assessments. 

APS Proposed R23. Each Balancing Authority shall have data exchange capabilities, with redundant and diversely routed data exchange infrastructure used to maintain its operations within the Balancing Authority's primary Control Center, for the exchange of Real-time data with its Reliability Coordinator, Transmission Operator, and the entities it has identified it needs data from in order for it to perform its Real-time monitoring and analysis functions. 

Michelle Amarantos, APS - Arizona Public Service Co., 5, 10/12/2016

- 0 - 0

Comments: See comments below.

- 0 - 0

R10.4, R10.5, & R10.6 duplicate the requirements in the already approved TOP-003-3, R1.

R19 & R20 overlap with the requirements in the already approved TOP-003-3, R5.

Colorado Springs Utilities, Segment(s) 1, 6, 3, 5, 9/14/2015

- 0 - 0

R10.3 states that the TOp must monitor non-BES facilities within its Transmission Operator Area identified as necessary by the Transmission Operator.  In the latest revision of the standard, the Standard Drafting Team added language to the Rationale section of the draft standard to clarify that the non-BES facilties that the TOp is required to monitor are “only those that are necessary for the TOp to determine SOL exceedances within its Transmissoin Operator Area.”  While TVA appreciates the additional details in the Rationale section, we feel that the additional details around identification of non-BES facilities belongs in the requirement itself.  Compliance obligations are now spreading down below 100 kV facilities into the non-BES system.  Whereas there use to be a hard line between where the compliance purview began and stopped, it will now be at times, more ambiguous.  For example, now non-BES facilities can be labeled BES facilities or, if this standard passes as written, non-BES facilities will still be non-BES but be required to be monitored if the TOp decides as such.  We would prefer to see more clarity in the requirement as to when the TOp would require a non-BES element be monitored.  TVA suggests the following language change for the requirement:

 

“R10.3 Monitor non-BES facilities within its Transmission Operator Area identified as necessary by the Transmission Operator in order to determine SOL exceedances on BES elements whithin its Transmission Operator Area.

Joel Wise, On Behalf of: Tennessee Valley Authority - SERC - Segments 1, 3, 5, 6

- 0 - 0

NVE still has concerns with the identification of non-BES facilities.  The proposed TOP-001-4 RSAW requires the Transmission Operator to provide evidence that it monitored all the data for non-elements identified, but no guidance on evidence to show that we do or do not have non-BES facilities.  Would we need to provide studies to show that we have no non-BES facilities?  We also have concerns about how non-BES facilities outside the Transmission Operators Area should be identified by the TOP.  Some sort of methodology or guidance should be added to the Requirement to demonstrate the identification of non-BES facilities.

- 0 - 0

R16, R17:  See our comment regarding R4 of IRO-002-5 above.

R20-R24:  Redundant infrastructure and redundant functionality terms are again used in different requirements. An entity can have redundant functionality without redundant infrastructure.  For example, you can have a single router/switch/piece of network equipment with redundant paths/functionality, but if power to the switch is lost then the functionality is lost because you don’t have redundant infrastructure.

R21: It is not clear what is required to test redundant functionality.  This could include each piece of network infrastructure inside the primary control center (ICCP boxes, routers, switches, EMS computers, System Operator computer consoles, etc.)  If this is left vague then it is not a good fit for a standard, but should be considered a guideline. 

R21-24: We request further clarification/explanation from the drafting team on the extent of the testing addressed in R21. Is it the drafting team’s intent to require an entity to test an entire pathway for redundant functionality every 90 calendar days, or is testing required on single elements only? The language in the requirement does not specifically address the extent of the testing expected. We recommend that language clearly outlining the extent of testing necessary to achieve compliance necessary be inserted in the requirement(s) or perhaps further explanantion in the rationale.

Also, Duke Energy is unsure that the proposed requirements R20-R24, do not fit with the overall purpose of the TOP standard family. The purpose outlined in TOP-001-4 states:

“To prevent instability, uncontrolled separation, or Cascading outages that adversely impact the reliability of the Interconnection by ensuring prompt action to prevent or mitigate such occurrences.”

We do not believe that the required actions in R20-R21 and R23-R24 are placed appropriately in this standard.  We are in agreement that some of the actions may be deemed necessary, however, we are not convinced that said actions would prevent instability, uncontrolled separation, or Cascading outages that may adversely impact reliability of an Interconnection. Duke Energy suggests the drafting team consider another standard, perhaps TOP-003-3, for the directed requirements.  TOP-003-3 aligns better with these new requirements instead of being placed in TOP-001-4 and suggest moving R19 and R22 from TOP-001-4 to TOP-003-3 also (these requirements are currently R19 and R20 in TOP-001-3).

Duke Energy , Segment(s) 1, 5, 6, 4/10/2014

- 0 - 0

ATC is concerned regarding requirement 10.3 as there is a perceived disconnect between the TOP requirement to monitor without a corresponding requirement for non-registered entities to provide requested data needed for monitoring.  The standard as written requires the TOP to monitor non-BES facilities within its Transmission Operator Area.  In one specific case in our system the entity who owns the facilities and thus manages the model and real time data is not a registered TOP, BA, GO, GOP, LSE, TO, or DP so they have no compliance obligation to provide the data.  As good utility practice we believe they should provide the data but that’s no guarantee that they will.  If ATC as the TOP does not have the correct oeprating parameters, whether impedances, charging values or ratings, or we do not have the correct telemetry, we cannot monitor their facilities (e.g., confirm flows are within limits). If we cannot monitor, we cannot be compliant.

Consider amending R10.3 to read as follows:

Monitor non-BES facilities within its Transmission Operator Area identified as necessary by the Transmission Operator.  In those cases where sufficient modeling and real time data is not available from the facility owner and they are not required to provide it monitoring is not feasible and thus not required.

 Requirement 20 was modified to indicate the need for redundant and diversely routed data exchange capabilities within the TOPs primary control center.  The way the standard is written ignores the enhanced redundancy ATC has implemented between control centers such that we can survive the loss of a single control center.  I believe it is contrary to the intent of the commission order (see below) which requires that “the data exchange capabilities of the transmission operators and balancing authorities require redundancy and diverse routing”.  I would recommend that the SDT modify the wording to allow for TOPs or BAs that have implemented redundancy across multiple primary control centers.

Requirement 21 was modified such that data exchange capabilities used in the primary Control Center have to be tested once every 90 days.  The SDT did extend that from every 30 days which was a positive result of the first round of comments.  Since ATC’s redundancy is built into the overall system architecture, where the loss of an entire control center can be withstood, verifying capabilities within one or both of our centers is above and beyond the intent of the FERC order (attached).

 

 

ATC agrees with the following comments put forth by NERC Stansdards Review Forum (NSRF).

 

Suggestions for Requirements R20 and R23:

The term ‘redundant and diversely’ routed is undefined and ambiguous. NSRF suggests the following wording change for these two requirements: “Each Balancing Authority (TOP) shall have data exchange capabilities, with redundant and diversely routed that reduce or mitigate single points of failure of data exchange infrastructure within the Balancing Authority's(TOP) primary Control Center in the exchange of identified Real-time data” .  The NSRF believes that this suggested language can also be applied to other Entities that require a reduction of single points of failure.

 

Suggestions for R21 and R24:

The NSRF thanks the drafting team for changing the periodicity for testing data exchange capabilities from the previous monthly periodicity. Within the comments to the first round of comments the drafting team indicated that they had moved the periodicity to quarterly, but actually put in 90 days. This may sound like a small thing, but from a compliance standpoint tracking 90 day periodicity versus a quarter periodicity can be a big thing. If an entity tracks by a quarter and completes their testing on a 91st day of a 91 day quarter, they are out of compliance. In addition, our technicians that test would find it less of a compliance burden to track quarter testing than tracking 90 day intervals. We would suggest that the wording in these two requirements actually use the term “quarter” or “quarterly”.

Andrew Pusztai, 10/14/2016

FERC Order.jpg

- 0 - 0

TOP-001-4 R20 - Please see comments regarding data exchange capabilities noted in Question #1.

Southern Company, Segment(s) 1, 3, 5, 6, 8/3/2016

- 0 - 0

Requirement R20:

NSRF revisions add further clarity to redundant and diversely routed within the Requirement.  It’s important to maintain a balance between being specific and overly prescriptive in a mandatory zero-defect environment.  We suggest the following revision allows entities to clearly define two primary control center data paths and the flexibility to identify what needs to be redundant while meeting FERC’s objectives.  Allowing entities to define two data infrastructure paths also recognizes that auditing to all possible single points of failure isn’t realistic or feasible. 

R20. Each Transmission Operator shall have data exchange capabilities, with redundant (meaning at least two data exchange paths exist for normal operating conditions) and diversely routed data exchange infrastructure (meaning switches, routers, file servers, power supplies, and network cabling in communication paths between these components in the two identified data exchange paths) within the Transmission Operator's primary Control Center, for the exchange of Real-time data with its Reliability Coordinator, Balancing Authority, and the entities it has identified it needs data from in order for it to perform its Real-time monitoring and Real-time Assessments. The planned or unplanned loss of one of the two data exchange infrastructure paths does not require further actions to meet compliance.

Requirement R23:

The following suggested revisions makes R23 consistent with the revisions suggested for R20.

R23. Each Balancing Authority shall have data exchange capabilities, with redundant and diversely routed data exchange infrastructure (meaning switches, routers, file servers, power supplies, and network cabling in communication paths between these components in the two identified data exchange paths) within the Balancing Authority's primary Control Center, for the exchange of Real-time data with its Reliability Coordinator, Transmission Operator, and the entities it has identified it needs data from in order for it to perform its Real-time monitoring and analysis functions.  

Suggestion for R21 and R24:

The NSRF thanks the drafting team for changing the periodicity for testing data exchange capabilities from the previous monthly periodicity. NSRF recommends changing the revised 90-day testing periodicity to quarterly.  While the industry appreciates standardization, there is a benefit to changing 90 days to quarterly.  This avoids continually accelerated tracking and rotating compliance periods. Operationally, moving to the largest testing period to maintain adequate reliability allows personnel flexibility in scheduling, tracking, and completing work.  Quarterly is superior to 90-days, Bi-annual is superior to 6-months, and annual is best unless a specific reliability need is identified.

MRO-NERC Standards Review Forum (NSRF), Segment(s) 3, 4, 5, 6, 1, 2, 7/14/2016

- 0 - 0

TOP-001-4 Requirement R10 is redundant with IRO-002-5requirmeent R5.  Both refer to the moinitoring of facilities.  The revisions to the Rationale for Requirement R10 reinforce this.

RSC no National Grid, Segment(s) 1, 0, 2, 4, 5, 6, 7, 3, 10/14/2016

- 0 - 0

Karl Blaszkowski, CMS Energy - Consumers Energy Company, 3, 10/14/2016

- 0 - 0

Karl Blaszkowski, CMS Energy - Consumers Energy Company, 3, 10/14/2016

- 0 - 0

Karl Blaszkowski, CMS Energy - Consumers Energy Company, 3, 10/14/2016

- 0 - 0

Karl Blaszkowski, CMS Energy - Consumers Energy Company, 3, 10/14/2016

- 0 - 0

<p>Please see Texas RE&rsquo;s response to the #1 regarding the specificity of primary control centers.&nbsp; The same concerns apply to IRO-002-5.</p><p>&nbsp;</p><p>Texas RE noticed TOP-001-4 Requirements R19 and R22 do not specify &ldquo;data exchange capabilities, with redundant and diversely routed data exchange infrastructure&rdquo;.&nbsp; Requirements R20 and R23 do specify &ldquo;data exchange capabilities, with redundant and diversely routed data exchange infrastructure&rdquo;.&nbsp; Is it the SDT&rsquo;s intent that the data TOPs and BAs use to develop an Operations Planning Analysis not be redundant and diversely routed?</p>

Rachel Coyne, Texas Reliability Entity, Inc., 10, 10/14/2016

- 0 - 0

- 0 - 0

We agree with the overall objective of the proposed revisions to the standard to ensure accurate system modeling for Real-time Assessment and real-time monitoring along with driving towards reducing the impacts to those processes of single points of failure for data systems.  However, the proposed verbiage changes from the previously balloted standard concerning testing frequency in R21 and R24 does not go far enough to differentiate between situations where redundant internal data exchange capability is provided in an active-active configuration versus an active-standby configuration.  In an active-active configuration the redundant internal data exchange capability is being tested through the ongoing use and monitoring of the equipment providing the redundant capability.  In an active-standby configuration the equipment providing the redundant internal data exchange capability is not being continually tested and warrants an explicit testing requirement.  The quarterly testing requirement for an active-standby configuration is appropriate.  In an active –active configuration no dedicated testing is necessary at any scheduled intervals since equipment is continually being tested through use and monitoring.  This approach encourages an active-active configuration which clearly provides enhanced reliability since there are no potential gaps where redundant capability is lost and not recognized until the next test.

- 0 - 0

We would like to see some changes in the Rationale for R10 to clarify that the intent is to monitor the non-BES facilities ‘so that a TOP can determine SOL exceedances’ not just monitor non-BES facilities.  We think clarifying also that the reliability impact to be guarded against is impact on the BES, not on the non-BES facilities.  Please make the following change:  The intent of the requirement is to ensure that all facilities (i.e., BES and non-BES) that can adversely impact BES reliability are monitored.

A similar change would also be helpful in the following sentence:

The non-BES facilities that the TOP is required to monitor are only those that are necessary for the TOP to determine SOL exceedances on BES Facilities within its Transmission Operator Area.

What does the SDT consider “data exchange infrastructure”?  Without an understanding of the intent of this language, it is unclear where the expectation for “redundant and diversely routed” ends.  If the intent is to require the same level of demonstration of evidence as was provided under the old COM-001-1, then redundancy typically only had to be demonstrated by showing the two separate telecomm lines going to two separate routers and then from there it went into the single firewall and then into the ESP.  If the ‘primary Control Center’ is considered within the single firewall/ESP boundary, then that should be clarified further in the requirement. 

Suggested changes to R2:

R20. Each Transmission Operator shall have data exchange capabilities, with redundant and diversely routed data exchange infrastructure after the point the data enters the Transmission Operator's primary Control Center, for the exchange of Real-time data with its Reliability Coordinator, Balancing Authority, and the entities it has identified it needs data from in order for it to perform its Real-time monitoring and Real-time Assessments.

We also would like to see the 90 day requirement to test changed to ‘Quarterly’. 

We have potential concerns (related to ‘where’ the boundary is considered for the Control Center) about which components need to be tested and what is considered an adequate test.  Without knowing what components are included we may not test the right things.

SPP Standards Review Group, Segment(s) , 10/14/2016

- 0 - 0

FMPA believes the meaning of redundant and diversely routed remains unclear, and that entities (and auditors) would benefit from having some examples of configurations that meet the expectations. Does a failover configuration where there is a potential for multiple combinations of active (or live) components meet the redundancy and diversely routed requirement, or does it require a completely separate set of isolated components from top to bottom? For example, a primary server could be setup to be capable of using either a primary and secondary switch. The secondary server would be setup the same way, so at any given time a combination of primary and secondary devices could be active. The boundary of what is considered within the Control Center is also unclear.

Entities are currently having to decipher what is required or proposed to be required by the CIP standards, which involve the very same equipment being discussed here. It is vital that the various Subject Matter Experts involved in the two efforts speak the same language and have a common understanding of what is meant by words such as “failure or malfunction” and “redundant and diversely routed”.  We believe there is too much room for interpretation as the Requirements are currently worded.

FMPA, Segment(s) , 8/3/2016

- 0 - 0

Scott Downey, 10/14/2016

- 0 - 0

Jerome Gobby, 10/14/2016

- 0 - 0

(1)   We thank the SDT for responding to our request to clarify that testing of data exchange capabilities should only apply to the primary Control Center and at a required frequency greater than monthly.

(2)   We caution the SDT in its phrasing of language used to address a FERC directive requiring TOPs to monitor non-BES facilities, as deemed necessary by the TOPs, to fill their functional obligations.  Rather than dive into a philosophical discussion regarding States rights versus the jurisdiction of FERC, we focus our concerns on the practical application of this language.  Many of the non-BES facilities are owned and maintained by entities not listed within the NERC compliance registry.  Some of these non-registered entities were de-registered following the approval of the Risk-based Registration initiative.  Moreover, owners of non-BES facilities outside a TOP Area may not have direct business ties or incentives to coordinate with the TOPs.  We recommend removing non-BES facility references from these standards, or as an alternative, rephrasing the appropriate parts of R10 to monitor and obtain statuses, voltages, and flow data for non-BES facilities, when such information is available.

(3)   Moreover, we continue to have concerns that the proposed additional requirements require a registered entity to possess data exchange capabilities and not maintain such capabilities.  By focusing on possession, a registered entity could instantly become non-compliant if these data exchange capabilities or associated analytical tools become unavailable.  We believe the requirements should follow a more performance-based approach and utilize the associated VSLs to identify the severity of non-compliance.  For instance, we propose rewording Requirement R19 to “Each TOP shall maintain data exchange capabilities with entities it deems necessary to perform its Operational Planning Analyses.”  This proposal could be reused to modify the similar BA requirement, R22.

(4)   Likewise, we propose rewording Requirement R20 to “Each Transmission Operator shall maintain data exchange capabilities, with redundant and diversely routed data exchange infrastructure within the Transmission Operator's primary Control Center, needed to perform Real-time monitoring and Real-time Assessments.”  This proposal could be reused to modify the similar BA requirement, R23.

(5)   We feel compliance should be embedded within existing business processes to better adopt such practices within a registered entity’s operations.  Many registered entities already execute or follow the execution of quarterly processes, and we believe the testing of data exchange capabilities could be included in such processes like quarterly model updates.  The tracking of every 90 days could be cumbersome for registered entities to coordinate test schedules and staffing levels for adequate test participation in advance.  Moreover, it may be possible that two tests are conducted within the same quarter, something the SDT is likely trying to avoid, and could fall during operating periods that are of high risk to the BES.   We recommend the periodicity of these tests, as identified in R21 and R24, be changed to calendar quarters

(6)   We believe the use of the NERC-defined Glossary Term, Operating Plan, is incorrectly applied in Requirement R22.  To paraphrase, an Operating Plan is a group of activities, Operating Procedures, or Operating Processes that are used to achieve a goal.  In the case of this requirement, what is the goal a BA assessing its next-day operations trying to achieve?  We recommend avoid using the NERC glossary term in this context or use a NERC defined term like “Adequacy.”

(7)   In light of the removal of operating logs as evidence identified within Measures M20 and M23, we ask the SDT to reflect this removal in the Evidence Retention Section of this standard (i.e. Section C.1.2).

ACES Standards Collaborators, Segment(s) 1, 3, 5, 6, 4, 10/14/2016

- 0 - 0

For R9, the focus on this comment is based on how this relates to EMS, SCADA and associated control systems.  WAPA would contend that switching to a redundant host (server, system, computer, etc) that provides functionally equivalent service, that this would not fall under the banner of a “planned outage”.  If an entity were to not have functionally equivalent redundant hosts and perform a switch-over, this would fall under the planned or unplanned outage banner.  WAPA would like to get clarification on this as to plan appropriately for its process to meet the new standard verbiage.

 

For R20, The verbiage that was changed in the rationale raises concerns as it relates to how this will be audited.  Data Exhcange Infrastructure is used in both the Rationale, the Requirement, and the Measure.  Yet the Rationalse provides signinficantly more detail and yet at the same time a wider scope.  The concern is focused on the items listed in the Rationale:

 

(switches, routers, file servers, power supplies, and network cabling and communication paths between these components in the primary Control Center for the exchange of system operating data)

 

The first example of concern with this verbiage is that the Requirement and the Standard focuses on “redundant and diversely routed data exchange infrastrcuture” but the Rationale seems to expand the scope of this to focus on components of devices instead of the devices themselves.  To draw a logical leap, would an entity be expected to have devices with redundant power supplies or is redundant power into the control center sufficient.  WAPA would like to see the Rationale match the Standard and Measure verbiage.  The other seemingly untouched area is the discussion around technology that make redundant paths much less effective or possibly not needed at all.  If, for example, a group of entities were to have a cloud technology (lets use MPLS for example) infrastructure that provides redundancy; would this meet the letter of the law even though redundancy at a device level may not exist?  This may not necessarily meet compliances as described but provides the level redundancy that the standard is striving for.  WAPA feels the focus on redundant and diverse links may cause the industry to miss the wider use of many technogologies available to us because we would be focused on meeting compliance rather than engineering a reliable and effective solution.

sean erickson, Western Area Power Administration, 1, 10/14/2016

- 0 - 0

Please see comments in response to Question #5.

Elizabeth Axson, 10/14/2016

- 0 - 0

Cain Braveheart, On Behalf of: Bonneville Power Administration, WECC, Segments 1, 3, 5, 6

- 0 - 0

No.

The SDT revisions to the R10 Rational do not address the double criteria application created by the proposed revisions to Requirement 10.

The SDT, in response to TOP-001-4 R10 Draft 1 comments, writes, “The SDT does not agree that the proposed changes to R10 affect the applicability of facilities within the CIP-002-5.1 standards.”

Without understanding how the SDT came to their position, we have to respectfully disagree with the SDT’s assessment.

The issue is not R10 affecting the applicability of facilities within CIP-002-5.1; the issue is R10 may create compliance obligations under CIP-002-5.1.

When a non‐BES Facility can adversely impact reliability is identified under R10; it essentially becomes a BES Facility (“Converted non‐BES Facility”, our term for purposes of these comments). The STD’s comments confirm this view, writing, “The SDT agrees that analyses performed in support of BES inclusions can identify some non-BES facilities that should be monitored for reliability…“

The Converted non‐BES Facility now is treated as a BES Facility, falling within the CIP‐002‐5.1 Applicability Sec. 4.2.2., “All BES Facilities.”

As a BES Facility, Entities are required to evaluate the Facility under CIP-002 to determine whether it is a High, Medium, or Low Impact BES Cyber System.

Our Concern

Bringing the BES Facility into CIP‐002‐5.1 Applicability creates a double impact criteria situation.

In other words, not identifying the non-BES Facility as potentially impacting the BES under R10—a compliance failure—automatically creates another compliance failure under CIP-002-5.1 because all BES Facilities are to be categorized and, as required, protected.

Put another way, in the event a non-BES Facility impacting the BES is not identified under R10 and should have been—it was missed—the Entity would be hard-pressed to justify that CIP-002-5.1 only applies to the non-BES Facilities identified under R10 and not to the “missed” Facilities. It would have to be a common sense justification but from a real-world view, the proposition is difficult to defend.

The current Proposed Standard creates the situation that a compliance failure with R10 creates a compliance failure under CIP-002-5.1.

Suggestion to Address the Issue

We do not believe this is an instance when the issue can be addressed by a single SDT; it requires the CIP Modifications SDT and potentially others, current and the future, to consider how revisions align with other Standards and the potential for a double impact criteria situation.

Douglas Webb, 10/14/2016

- 0 - 0

Hot Answers

Thomas Lyons, Owensboro Municipal Utilities, 3, 10/17/2016

- 0 - 0

Daniel Herring, DTE Energy - Detroit Edison Company, 4, 10/17/2016

- 0 - 0

Other Answers

Michael Puscas, ISO New England, Inc., 2, 9/21/2016

- 0 - 0

- 0 - 0

Thomas Foltz, AEP, 5, 10/4/2016

- 0 - 0

Diana McMahon, On Behalf of: Salt River Project, WECC, Segments 1, 3, 5, 6

- 0 - 0

Jack Stamper, Clark Public Utilities, 3, 10/5/2016

- 0 - 0

Steven Rueckert, Western Electricity Coordinating Council, 10, 10/7/2016

- 0 - 0

Richard Vine, 10/10/2016

- 0 - 0

David Jendras, Ameren - Ameren Services, 3, 10/11/2016

- 0 - 0

- 0 - 0

Karie Barczak, On Behalf of: DTE Energy - Detroit Edison Company, , Segments 3, 4, 5

- 0 - 0

No - TOP-001-4

No - IRO-002-5

Scott McGough, Georgia System Operations Corporation, 3, 10/11/2016

- 0 - 0

Amy Casuscelli, On Behalf of: Xcel Energy, Inc. - MRO, WECC, SPP RE - Segments 1, 3, 5, 6

- 0 - 0

If requirement R10 allows the RC to stipulate non-BES facilities be monitored, 24 months should be allocated to install, test and implement the proper equipment.

In addition, if requirements R20 and R23 remain as they are currently worded, such that the TOP and/or BA are required to install redundant and diversely routed data exchange infrastructure within the primary Control Center (only), and do not allow for flexibility for the redundant and diversely routed data exchange infrastructure to span across Control Centers, then APS proposes a minimum of 3 calendar years for implementation.

Michelle Amarantos, APS - Arizona Public Service Co., 5, 10/12/2016

- 0 - 0

Comments: See comments below.

- 0 - 0

Colorado Springs Utilities, Segment(s) 1, 6, 3, 5, 9/14/2015

- 0 - 0

Joel Wise, On Behalf of: Tennessee Valley Authority - SERC - Segments 1, 3, 5, 6

- 0 - 0

Depending on which non-BES facilities are identified in Requirement 10, additional infrastructure may be required to bring back the necessary data identified in TOP-003-3 to monitor non-BES facilities.  In that scenario, more time may be needed than what is proposed for Requirement 10. 

- 0 - 0

Duke Energy has concerns regarding the effort to achieve compliance with IRO-002-4, and the possible quick turnaround for becoming compliant with IRO-002-5. Entities have already made interpretations, and taken specific actions to achieve compliance with IRO-002-4 prior to it being enforceable in April of 2017. With the potential for such a quick turnaround in standard versions, we believe that delaying the enforcement date for IRO-002-4 until IRO-002-5 is approved and enforceable would be prudent. The delaying of enforcement dates to consolidate versions has happened in the past (see PRC-005). We believe that rolling all changes and enforcement dates to the potential enforcement date for IRO-002-5 is a practical solution for industry stakeholders in achieving compliance with the different versions.

Duke Energy , Segment(s) 1, 5, 6, 4/10/2014

- 0 - 0

: No concerns with the timelines proposed with implementation assumed 4/1/2018 based on current schedule.

Andrew Pusztai, 10/14/2016

- 0 - 0

Please see comments for Question #1.

Southern Company, Segment(s) 1, 3, 5, 6, 8/3/2016

- 0 - 0

MRO-NERC Standards Review Forum (NSRF), Segment(s) 3, 4, 5, 6, 1, 2, 7/14/2016

- 0 - 0

RSC no National Grid, Segment(s) 1, 0, 2, 4, 5, 6, 7, 3, 10/14/2016

- 0 - 0

Karl Blaszkowski, CMS Energy - Consumers Energy Company, 3, 10/14/2016

- 0 - 0

Karl Blaszkowski, CMS Energy - Consumers Energy Company, 3, 10/14/2016

- 0 - 0

Karl Blaszkowski, CMS Energy - Consumers Energy Company, 3, 10/14/2016

- 0 - 0

Karl Blaszkowski, CMS Energy - Consumers Energy Company, 3, 10/14/2016

- 0 - 0

<p>Texas RE appreciates the SDT&rsquo;s inclusion of an &ldquo;Initial Performance of Periodic Requirements&rdquo; section.&nbsp; Texas RE believes that this is a best practice and recommends that future SDTs provide a similar section for all periodic requirements to avoid ambiguity.</p><p>Texas RE does not necessarily object to the SDT&rsquo;s proposed 12-month implementation period.&nbsp; However, Texas RE respectfully requests that the SDT provide a basis for its decision to adopt such a 12-month compliance window, including any data it considered in determining that this was an appropriate window for affected entities to meet their compliance obligations under the revised Standards.&nbsp;</p>

Rachel Coyne, Texas Reliability Entity, Inc., 10, 10/14/2016

- 0 - 0

- 0 - 0

- 0 - 0

We need further guidance on which aspects of our data exchange capability should be redundant in order to answer this question.  The old COM-001 was more focused on capabilities outside the Control Center (avoiding the ‘backhoe’ outages) while this requirement seems to be focused on redundancy within the data center, but ignoring redundancy outside the data center.  Additional rigor may need to be added to internal redundancy.  Based on that, the 12 month implementation plan may be insufficient.  Redundancy and diverse routing in the legacy requirements seemed to mean something different than is being presented in the directive by FERC today. 

SPP Standards Review Group, Segment(s) , 10/14/2016

- 0 - 0

FMPA, Segment(s) , 8/3/2016

- 0 - 0

Scott Downey, 10/14/2016

- 0 - 0

Jerome Gobby, 10/14/2016

- 0 - 0

We thank the SDT for providing clarity that the scope of the requirements are aimed at the applicable entity's primary Control Center.  However, we disagree with the SDT that entities will have sufficient time with a 12-month implementation plan.  That assumption is based on TOPs and BAs already have adequate infrastructure and data exchange capabilities in place to perform their SOL exceedance determinations, monitoring, and assessment calculations.  We believe there is a possibility that they won’t, particularly with owners of non-BES facilities that are identified as necessary for TOPs and BAs to complete their functional obligations.  In this instance, TOPs and BAs could be faced with a compliance-based decision to either sacrifice reliability concerns by identifying these facilities as not necessary for monitoring and assessments or identify, procure, install, and continue to maintain adequate infrastructure and data exchange capabilitibities with these facilities in order to remain compliant.  In the latter instance, incurring such costs could be done outside budgeting approvals for smaller entities, particularly in a compressed 12-month implementation period.  We propose a 24-month implementation period instead, as that could accommodate 2-3 possible budget cycles.

ACES Standards Collaborators, Segment(s) 1, 3, 5, 6, 4, 10/14/2016

- 0 - 0

sean erickson, Western Area Power Administration, 1, 10/14/2016

- 0 - 0

Elizabeth Axson, 10/14/2016

- 0 - 0

Cain Braveheart, On Behalf of: Bonneville Power Administration, WECC, Segments 1, 3, 5, 6

- 0 - 0

No.

The proposed Implementation Plan does not consider the time required to meld the technology required to address compliance under TOP-001-4 R10 and compliance under CIP-002-5.1. The period should be extended by a year.

Douglas Webb, 10/14/2016

- 0 - 0

Hot Answers

Thomas Lyons, Owensboro Municipal Utilities, 3, 10/17/2016

- 0 - 0

Daniel Herring, DTE Energy - Detroit Edison Company, 4, 10/17/2016

- 0 - 0

Other Answers

The language for the NERC Standard, IRO-002-5, R3, Severe VSL. The last paragraph states: The Reliability Coordinator tested its primary Control Center data exchange capabilities specified in Requirement R2 for redundant functionality at least once each every 90 calendar month days but, following an unsuccessful test, did not initiate action to restore the redundant functionality in more than 8 hours.

This does not convey the intent. Literally, it says that the violation occurred because the action was initiated in 8 hours or less or that there were 8 or fewer hours in which actions were initiated. I’d suggest changing the language to:

The Reliability Coordinator tested its primary Control Center data exchange capabilities specified in Requirement R2 for redundant functionality at least once each every 90 calendar month days but, following an unsuccessful test, did not initiate action to restore the redundant functionality within 8 hours of the test failure.

Michael Puscas, ISO New England, Inc., 2, 9/21/2016

- 0 - 0

We are concerned over the issue raised for "test" in TOP-001-4  Requirements R21 & R24 pertaining to the lack of clarity for implementation and audit approach and therefor cannot agree with and VRF/VSL associated with these requirements.

- 0 - 0

Thomas Foltz, AEP, 5, 10/4/2016

- 0 - 0

Diana McMahon, On Behalf of: Salt River Project, WECC, Segments 1, 3, 5, 6

- 0 - 0

Jack Stamper, Clark Public Utilities, 3, 10/5/2016

- 0 - 0

Steven Rueckert, Western Electricity Coordinating Council, 10, 10/7/2016

- 0 - 0

Richard Vine, 10/10/2016

- 0 - 0

David Jendras, Ameren - Ameren Services, 3, 10/11/2016

- 0 - 0

- 0 - 0

Karie Barczak, On Behalf of: DTE Energy - Detroit Edison Company, , Segments 3, 4, 5

- 0 - 0

No - TOP-001-4

No - IRO-002-5

Scott McGough, Georgia System Operations Corporation, 3, 10/11/2016

- 0 - 0

Amy Casuscelli, On Behalf of: Xcel Energy, Inc. - MRO, WECC, SPP RE - Segments 1, 3, 5, 6

- 0 - 0

Michelle Amarantos, APS - Arizona Public Service Co., 5, 10/12/2016

- 0 - 0

Comments: See comments below.

- 0 - 0

Colorado Springs Utilities, Segment(s) 1, 6, 3, 5, 9/14/2015

- 0 - 0

Joel Wise, On Behalf of: Tennessee Valley Authority - SERC - Segments 1, 3, 5, 6

- 0 - 0

- 0 - 0

Duke Energy , Segment(s) 1, 5, 6, 4/10/2014

- 0 - 0

Andrew Pusztai, 10/14/2016

- 0 - 0

Please see comments for Question #1.

Southern Company, Segment(s) 1, 3, 5, 6, 8/3/2016

- 0 - 0

MRO-NERC Standards Review Forum (NSRF), Segment(s) 3, 4, 5, 6, 1, 2, 7/14/2016

- 0 - 0

RSC no National Grid, Segment(s) 1, 0, 2, 4, 5, 6, 7, 3, 10/14/2016

- 0 - 0

Karl Blaszkowski, CMS Energy - Consumers Energy Company, 3, 10/14/2016

- 0 - 0

Karl Blaszkowski, CMS Energy - Consumers Energy Company, 3, 10/14/2016

- 0 - 0

Karl Blaszkowski, CMS Energy - Consumers Energy Company, 3, 10/14/2016

- 0 - 0

Karl Blaszkowski, CMS Energy - Consumers Energy Company, 3, 10/14/2016

- 0 - 0

Rachel Coyne, Texas Reliability Entity, Inc., 10, 10/14/2016

- 0 - 0

- 0 - 0

As proposed, the VRFs and VSLs treat each R10 sub requirement equally when in reality both the risk and resulting potential impact is significantly different between the requirements.  The risk and associated potential impact of a TOP not monitoring it’s own Facilities and RAS’s is significantly greater on the ability for state estimation and contingency analysis to solve and provide accurate results than to not monitor non-BES facilities within it’s system or external TOP areas’ Facilities/RAS’s/non-BES facilities. 

- 0 - 0

They seem consistent based on the requirements as stated.  If the periodicity changes to ‘quarterly’ the VRF’s and VSL’s would need to change accordingly

SPP Standards Review Group, Segment(s) , 10/14/2016

- 0 - 0

FMPA, Segment(s) , 8/3/2016

- 0 - 0

Scott Downey, 10/14/2016

- 0 - 0

Jerome Gobby, 10/14/2016

- 0 - 0

(1)   We thank the SDT for responding to our request to account for the “as necessary” parts of requirement R10 within the VSLs.

(2)   However, we believe the SDT has an opportunity to develop more performance-based requirements for these standards.  We believe the VSLs for these requirements should base compliance on a definite period of time that has lapsed when a registered entity is unable to perform its monitoring functions or conduct its assessments.  This scalable time duration could then be used to identify VSLs for the complete set, and not just values for Severe VSLs.  We propose an exceedance of 30 minutes listed as a Low VSL, with a 30-minute increment for each increasing severity limit.

ACES Standards Collaborators, Segment(s) 1, 3, 5, 6, 4, 10/14/2016

- 0 - 0

sean erickson, Western Area Power Administration, 1, 10/14/2016

- 0 - 0

Elizabeth Axson, 10/14/2016

- 0 - 0

Cain Braveheart, On Behalf of: Bonneville Power Administration, WECC, Segments 1, 3, 5, 6

- 0 - 0

Douglas Webb, 10/14/2016

- 0 - 0

Hot Answers

We continuously monitor our system and the assessment provides no benefit just additional administrative work.

Thomas Lyons, Owensboro Municipal Utilities, 3, 10/17/2016

- 0 - 0

Daniel Herring, DTE Energy - Detroit Edison Company, 4, 10/17/2016

- 0 - 0

Other Answers

Thanks for all your work.

Michael Puscas, ISO New England, Inc., 2, 9/21/2016

- 0 - 0

- 0 - 0

Thomas Foltz, AEP, 5, 10/4/2016

- 0 - 0

Diana McMahon, On Behalf of: Salt River Project, WECC, Segments 1, 3, 5, 6

- 0 - 0

Clark believes that R21 is still not addressing the issue FERC has expressed its concern over. Please read the paragraph in question. In Paragraph 51, Order 817, FERC states:

“We agree with NERC and other commenters that there is a reliability need for the reliability coordinator, transmission operator and balancing authority to test alternate data exchange capabilities. However, we are not persuaded by the commenters’ assertions that the need to test is implied in the TOP and IRO Standards. Rather, we determine that testing of alternative data exchange capabilities is important to reliability and should not be left to what may or may not be implied in the standards. Therefore, pursuant to section 215(d)(5) of the FPA, we direct NERC to develop a modification to the TOP and IRO standards that addresses a data exchange capability testing framework for the data exchange capabilities used in the primary control centers to test the alternate or less frequently used data exchange capabilities of the reliability coordinator, transmission operator and balancing authority. We believe that the structure of Reliability Standard COM-001-2, Requirement R9 could be a model for use in the TOP and IRO Standards.”

The only reference to the primary control center in Paragraph 51 is that the overall data exchange capabilities needs to be tested to ensure that “the alternate or less frequently used data exchange capabilities of the reliability coordinator, transmission operator and balancing authority” are tested. If the exchange capabilities used in the primary control centers are the primary or most frequently used data exchange capabilities, it is obvious that some other alternate data exchange is the scope of the FERC’s directive. This may be and alternate data exchange capability in the primary control center but there is nothing in Paragraph 51 that would preclude the use of a backup control center for the provision of the alternate or less frequently used data exchange capabilities. The SDT continues to INCORRECTLY apply this paragraph first to control centers and now to primary control centers. Paragraph 51 is applicable to the registered the entities RCs, TOPs, and BAs and requires these registered entities to test the alternate or less frequently used data exchange capabilities of the reliability coordinator, transmission operator and balancing authority. The SDT continues to fail to recognize that many RCs, TOPs, and BAs already have redundancy and diverse routing of data exchange that addresses the FERC’s concerns because they have voluntarily adopted this approach in meeting their COM standards compliance and their EOP-008 compliance. For these entities, testing the alternate data exchange capabilities of their backup control centers would address the FERC’s concerns expressed in Paragraph 51.

As such the SDT needs to modify R21 and M21 to the following:

“R21. Each Transmission Operator shall test its alternate or less frequently used data exchange capabilities specified in Requirement R20 for redundant functionality at least once each every 90 calendar days. If the test is unsuccessful, the Transmission Operator shall initiate action within two hours to restore redundant functionality. [Violation Risk Factor: Medium ] [Time Horizon: Operations Planning]”

“M21. Each Transmission Operator shall have, and provide upon request, evidence that it tested its alternate or less frequently used data exchange capabilities specified in Requirement R20 for the redundant functionality, or experienced an event that demonstrated the redundant functionality; and, if the test was unsuccessful, initiated action within two hours to restore redundant functionality as specified in Requirement R21. Evidence could include, but is not limited to: dated and time-stamped test records, operator logs, voice recordings, or electronic communications.”

The above revisions to R21 and M21 will cause Transmission Operators to test their alternate or less frequently used data exchange capabilities specified in Requirement R20 every 90 calendar days which is the concern FERC has expressed. These changes will allow TOPs to submit evidence that the alternate or less frequently used data exchange capabilities being voluntarily used in compliance with the COM standards and EOP-008-1 have been tested to demonstrate redundant functionality (as well as alternate or less frequently use data exchage capabilities at the primary control center). The SDT should have no concern that the FERC would reject this since Paragraph 51 clearly ONLY REQUIRES THE TESTING OF THE ALTERNATE OR LESS FREQUENTLY USED DATA EXCHANGE CAPABILITIES OF THE RCs, TOPs, and BAs (regardless of the location).

While Clark is not an RC or a BA, it believes the above arguments are also applicable to these registered entities and that any similar requirements and measures applicable to these entities in IRO-002-5 or TOP-001-4 should be similarly modified.

Jack Stamper, Clark Public Utilities, 3, 10/5/2016

- 0 - 0

Steven Rueckert, Western Electricity Coordinating Council, 10, 10/7/2016

- 0 - 0

Richard Vine, 10/10/2016

- 0 - 0

David Jendras, Ameren - Ameren Services, 3, 10/11/2016

- 0 - 0

- 0 - 0

Karie Barczak, On Behalf of: DTE Energy - Detroit Edison Company, , Segments 3, 4, 5

- 0 - 0

For IRO-002-5

 

Comments:   Southern believes that the language in IRO-002-5 R2 explicitly limits the scope of the requirement to “the data exchange infrastructure inside the Primary Control Center”. The first problem here is that the term “data exchange infrastructure” has no clear or broadly accepted industry definition.  

The second problem is that here is no clear definition of what constitutes the control center. Is it a facility or a room inside a facility? What prevents someone from moving the “capability” outside the control center (i.e a data center not part of the control center)?

 

The language in IRO-002-5 R3 currently has a requirement to test the “primary control center data exchange capabilities” specified in R2” every 90 days. First of all, the terminology shifts from the word “infrastructure” in R2 to “capabilities” in R3, which leaves a lot of ambiguity. Why establish a requirement to test the redundancy of the data exchange and not the EMS platform in which the capability resides.  This is perplexing given the fact that the data exchange function is, in most cases, a sub-component of the much larger distributed EMS architecture.

 

The language in IRO-002-5 R6 is also confusing,  as it states that  “each RC shall have monitoring systems that provide information utilized by the RC’s operating personnel, giving particular emphasis to alarm management and awareness systems, automated data transfers, and synchronized information systems, over a redundant infrastructure.  Again, this is very confusing, because the following questions have not been answered:

 

  • What is meant by “particular emphasis”?
  • Which “awareness systems” require redundant infrastructure
  • Which “automated data transfers” are in scope?
  • Which “synchronized information”  is in scope?
  • What level of redundancy is required? Server level, component level, network level, etc.

 

 

Scott McGough, Georgia System Operations Corporation, 3, 10/11/2016

- 0 - 0

Amy Casuscelli, On Behalf of: Xcel Energy, Inc. - MRO, WECC, SPP RE - Segments 1, 3, 5, 6

- 0 - 0

Michelle Amarantos, APS - Arizona Public Service Co., 5, 10/12/2016

- 0 - 0

Comments:   Southern believes that the language in IRO-002-5 R2 explicitly limits the scope of the requirement to “the data exchange infrastructure inside the Primary Control Center”. The first problem here is that the term “data exchange infrastructure” has no clear or broadly accepted industry definition.  The second problem is that here is no clear definition of what constitutes the control center. Is it a facility or a room inside a facility? What prevents someone from moving the “capability” outside the control center (i.e a data center not part of the control center)?

The language in IRO-002-5 R3 currently has a requirement to test the “primary control center data exchange capabilities” specified in R2” every 90 days. First of all, the terminology shifts from the word “infrastructure” in R2 to “capabilities” in R3, which leaves a lot of ambiguity. Why establish a requirement to test the redundancy of the data exchange and not the EMS platform in which the capability resides.  This is perplexing given the fact that the data exchange function is, in most cases, a sub-component of the much larger distributed EMS architecture.

The language in IRO-002-5 R6 is also confusing,  as it states that  “each RC shall have monitoring systems that provide information utilized by the RC’s operating personnel, giving particular emphasis to alarm management and awareness systems, automated data transfers, and synchronized information systems, over a redundant infrastructure.  Again, this is very confusing, because the following questions have not been answered:

  • What level of redundancy is required? Server level, component level, network level, etc.

  • Which “synchronized information”  is in scope?

  • Which “automated data transfers” are in scope?

  • Which “awareness systems” require redundant infrastructure

  • What is meant by “particular emphasis”?

- 0 - 0

Colorado Springs Utilities, Segment(s) 1, 6, 3, 5, 9/14/2015

- 0 - 0

Joel Wise, On Behalf of: Tennessee Valley Authority - SERC - Segments 1, 3, 5, 6

- 0 - 0

- 0 - 0

Duke Energy , Segment(s) 1, 5, 6, 4/10/2014

- 0 - 0

ATC agrees with the following comments put forth by NSRF.

Suggestions for the Rationale box for R20/R23:

For the paragraph that reads “The reliability objective of redundancy is to provide….” The current wording is not clear for the last sentence of the paragraph. The NSRF recommends that the last sentence of that paragraph be changed  to read “For periods of planned or unplanned outages of individual components in the primary or redundant data exchange path, the requirements do not require additional data exchange infrastructure or components during those planned and unplanned outage periods”.

The NSRF would like to point out the FERC Order 693, section 253 states that “…compliance will in all cases be measured by determining whether a party met or failed to meet the Requirement…”.  Within TOP-001-4 there are Rational boxes (e.g. R23) that explain in detail what Redundant and Diversely routed MAY mean.  Without these details being within the Requirement, “Redundant and diversely routed” become ambiguous words that will allow an Entity to believe they mean one thing and an auditor may believe in something else.

The NSRF recommends that the details written in the Rational box be prefaced with “Some examples of Redundant and diversely routed may mean … depending on how the responsible entity wishes to address their Redundant and diversely routed risks within their Primary Control Center”. 

 

Andrew Pusztai, 10/14/2016

- 0 - 0

Southern Company, Segment(s) 1, 3, 5, 6, 8/3/2016

- 0 - 0

The NSRF would like to point out that there is a great deal of compliance information in the newly updated Rationale Boxes.  We agree this gives insight to meeting the task(s) assigned to each Requirement, but does not allow for clear understanding to the applicable Entity and CEA Staff.  FERC Order 693, Section 253 states that applicable Entities must meet the word of the Requirement in order to show that they are Compliant with said Requirement.  The NSRF recommends that the intent of the Rationale Box be within each Requirement.

MRO-NERC Standards Review Forum (NSRF), Segment(s) 3, 4, 5, 6, 1, 2, 7/14/2016

- 0 - 0

RSC no National Grid, Segment(s) 1, 0, 2, 4, 5, 6, 7, 3, 10/14/2016

- 0 - 0

Karl Blaszkowski, CMS Energy - Consumers Energy Company, 3, 10/14/2016

- 0 - 0

Karl Blaszkowski, CMS Energy - Consumers Energy Company, 3, 10/14/2016

- 0 - 0

Karl Blaszkowski, CMS Energy - Consumers Energy Company, 3, 10/14/2016

- 0 - 0

Karl Blaszkowski, CMS Energy - Consumers Energy Company, 3, 10/14/2016

- 0 - 0

Rachel Coyne, Texas Reliability Entity, Inc., 10, 10/14/2016

- 0 - 0

- 0 - 0

- 0 - 0

N/A

SPP Standards Review Group, Segment(s) , 10/14/2016

- 0 - 0

FMPA, Segment(s) , 8/3/2016

- 0 - 0

None

Scott Downey, 10/14/2016

- 0 - 0

Jerome Gobby, 10/14/2016

- 0 - 0

(1)   We thank the SDT for its detailed information included in the standards’ rationale boxes, as such information is useful in understanding the purpose and intent of each requirement.  However, we caution that each requirement must be clear and understandable by both registered entities and auditors to demonstrate and measure compliance, respectively.  We recommend incorporating aspects of these rationale boxes, within the language of each requirement, where possible.

(2)   We thank the SDT for this opportunity to provide comments on these standards.

ACES Standards Collaborators, Segment(s) 1, 3, 5, 6, 4, 10/14/2016

- 0 - 0

For R9, the focus on this comment is based on how this relates to EMS, SCADA and associated control systems.  WAPA would contend that switching to a redundant host (server, system, computer, etc) that provides functionally equivalent service, that this would not fall under the banner of a “planned outage”.  If an entity were to not have functionally equivalent redundant hosts and perform a switch-over, this would fall under the planned or unplanned outage banner.  WAPA would like to get clarification on this as to plan appropriately for its process to meet the new standard verbiage.

sean erickson, Western Area Power Administration, 1, 10/14/2016

- 0 - 0

While ERCOT recognizes that the SDT added the word “primary” to further specify which control center is required to include redundant and diversely routed data exchange capabilities, ERCOT requests further clarification on the term “primary Control Center.”  While it can reasonably be assumed that “primary Control Center” is intended to refer to the Control Center normally used for daily, non-emergency “monitoring and control of the BES in real-time” rather than a Control Center normally used as a backup, this understanding should be explicitly stated to avoid any question as to the breadth of the standard’s scope.  

 

ERCOT asks for this clarification to ensure that a backup control center is not considered a “primary” control center during those temporary conditions in which a failover to the backup is required.  If an entity lacks redundant or diversely routed data exchange capabilities at a backup control center that may temporarily function as a “primary control center” for a brief period of time, auditors might deem the entity to be out of compliance for that duration.  ERCOT therefore recommends defining the term “primary Control Center” in the standard or in the NERC Glossary of Terms, as opposed to simply providing clarification in the rationale or measure, so that the clarification is more clearly enforceable. 

 

ERCOT notes that several other standards use the term “primary control center,” including EOP-008-1 – Loss of Control Center Functionality, and CIP-014-2 – Physical Security.  In CIP-014-2, the “Guidelines and Technical Basis” section states that the primary control center is “the control center that the Transmission Owner or Transmission Operator, respectively, uses as its primary, permanently-manned site to physically operate a Transmission station or Transmission substation…”   ERCOT suggests that a definition similar to that used in CIP-014-2 could provide the needed clarification.

 

 

100% Redundancy and Diversity Compliance at all times

 

As currently written, Requirement R2 of IRO-002-5 and requirement R23 of TOP-001-4, could still be read to require 100% system integrity at all times, no matter what events or failures may arise in an entity’s data exchange capabilities that could, for a time, cause the system to lack redundancy or diverse routing.

 

Is an entity out of compliance with these requirements if it has redundancy built into its system and, for the majority of the time, functions properly, but temporarily fails during a broader system failure?  Would an entity be considered non-compliant because the redundancy is temporarily unavailable?

 

While the SDT sought to address this concern by adding language to the rationale box and pointed to measure language regarding evidence, neither the rationale nor the measure is enforceable.  ERCOT therefore recommends adding the phrase “during normal system conditions” or “during normal system operations” to the identified requirements to clarify that entities are not required to have “additional redundant data exchange infrastructure components solely to provide for redundancy during planned or unplanned outages of individual components.”

 

Additionally, the SDT may wish to consider language similar to that used in EOP-008, R3:

“To avoid requiring tertiary functionality, backup functionality is not required during:

• Planned outages of the primary or backup functionality of two weeks or less

• Unplanned outages of the primary or backup functionality”

Elizabeth Axson, 10/14/2016

- 0 - 0

Cain Braveheart, On Behalf of: Bonneville Power Administration, WECC, Segments 1, 3, 5, 6

- 0 - 0

Kansas City Power and Light Company greatly appreciates the work of the Standard Drafting Team. Thank you.

Douglas Webb, 10/14/2016

- 0 - 0